This role sits at the intersection of advanced AI development and rigorous quality assurance within a leading technology organization focused on delivering secure, reliable solutions for mission-critical sectors. The Senior QA Automation Engineer will play a pivotal part in ensuring that innovative AI-driven systems meet the highest standards of reliability and compliance, supporting the organization’s commitment to operational excellence and technological advancement in highly regulated environments.

JD Romteca: Responsibilities

- Lead the design and implementation of automated testing frameworks for AI and software systems, focusing on both functional and AI-specific validation
- Develop and maintain automated test suites to evaluate Generative AI outputs for accuracy, consistency, and alignment with ground truth datasets
- Implement automated metrics and regression tests to monitor AI model performance, including prompt drift and retrieval-augmented generation accuracy
- Build and execute integration tests to ensure seamless data flow and consistency between AI agents and enterprise systems such as ERP platforms
- Design and run performance tests to benchmark forecasting models and risk scoring engines for latency and throughput requirements
- Automate API testing to verify secure data handling, role-based access control, and compliance with privacy requirements
- Map test cases to system requirements and user needs, ensuring traceability and supporting formal verification and validation processes
- Prepare and deliver test readiness documentation and evidence for stage gate and production release reviews
- Manage defect reporting and trace issues back to specific model versions or datasets, collaborating with development and data science teams to drive resolution

JD Romteca: Required Skills

- expert proficiency in Python for test automation (Pytest), UI automation (Selenium/Playwright), and API testing (Requests)
- advanced performance testing skills with tools such as Locust, JMeter, or K6; hands-on experience with LLM/AI evaluation frameworks (DeepEval, TruLens, or custom evaluators)
- strong SQL and data validation skills (Great Expectations or similar)
- experience integrating automated tests into CI/CD pipelines (GitLab preferred)
- familiarity with requirements traceability and test management tools (Jira, Jama, Polarion, TestRail, Allure)
- strong version control skills (Git, GitLab)
- deep understanding of modern quality engineering practices (Shift Left, Test Pyramid, mono-repo automation)
- ability to define and communicate pass/fail criteria for probabilistic AI systems
- experience in highly regulated or defense sectors is a plus
- strong analytical, collaboration, and communication skills.

How to recruit for this role?

When recruiting for this Senior QA Automation Engineer role, prioritize candidates with deep hands-on experience in Python-based automation, performance testing, and AI/LLM evaluation frameworks, especially those who have worked in highly regulated or safety-critical industries. Look for strong knowledge of integrating automated tests into CI/CD pipelines, mapping tests to requirements, and managing test artifacts and traceability. Focus on those who can demonstrate experience validating non-deterministic AI systems, not just traditional software, and who understand the unique challenges of testing probabilistic outputs and ensuring data integrity across complex integrations. Avoid candidates whose experience is limited to manual testing, basic UI automation, or who lack exposure to AI/ML systems and formal V&V processes. The most important quality is the ability to define and enforce rigorous, evidence-based quality standards for AI systems in mission-critical environments.

What's IMPORTANT for this job?

-

Client and Project information (can be shared confidentially)

The client is an engineering consulting company (similar to Accenture, IBM) from Sweden. They support other companies in developing software & hardware products. This particular project is developed for a client from UAE. The project details will be revealed in a technical interview, in case the client will want to meet the candidate. The work is remote 100%, the core team is in Abu Dhabi. Next steps are - CV submission -> technical screening by Romteca -> technical interview by the client-> decision -> onboarding.

Information about Romteca

Romteca is a recruitment agency for freelance engineering and IT projects. We work exclusively with freelance consultants and B2B contracts. We are based in Romania, have global clients and a global talent pool. We manage the whole cycle of recruitment, hiring, contracts, payments.