This role sits at the intersection of advanced AI development and rigorous quality assurance within a leading technology organization focused on delivering secure, reliable solutions for mission-critical sectors. The Senior QA Automation Engineer will play a key part in ensuring that innovative AI and supply chain systems meet the highest standards of reliability and compliance, supporting the organization’s commitment to operational excellence and technological advancement.

JD Romteca: Responsibilities

- Lead the design and implementation of automated testing frameworks for AI and LLM-based systems
- Develop and maintain automated test suites to validate AI outputs for accuracy, consistency, and alignment with ground truth datasets
- Build and execute integration tests to ensure seamless data flow between AI agents and enterprise systems such as ERP platforms
- Create and run performance tests to benchmark AI model latency, throughput, and reliability in real-time environments
- Automate API testing to verify secure data exchange, RBAC enforcement, and PII redaction before data reaches AI models
- Map automated test cases to system requirements and user needs to support formal verification and validation processes
- Prepare and deliver test readiness documentation and evidence for stage gate and production release reviews
- Manage defect identification, tracking, and resolution, ensuring traceability to specific model versions or datasets
- Collaborate with cross-functional teams, including data scientists and systems engineers, to align testing with acceptance criteria

JD Romteca: Required Skills

- Expert proficiency in Python for test automation
- Strong experience with Pytest, Selenium, and Playwright
- Advanced skills in performance testing tools (Locust, JMeter, K6)
- Hands-on experience evaluating LLMs and AI models (DeepEval, TruLens, custom evaluators)
- Proficiency in SQL and data validation tools (Great Expectations)
- Experience integrating automated tests into CI/CD pipelines (GitLab)
- Familiarity with requirements management and traceability tools (Jira, Jama, Polarion)
- Strong knowledge of test management and reporting tools (TestRail, Allure)
- Solid understanding of version control systems (Git, GitLab)
- Deep knowledge of modern quality engineering practices (Shift Left, Test Pyramid, mono-repo architecture)

How to recruit for this role?

When recruiting for this Senior QA Automation Engineer role, prioritize candidates with deep hands-on experience in Python-based automation, performance testing, and AI/LLM evaluation frameworks, especially those who have worked in highly regulated or safety-critical environments. Look for strong understanding of IV&V processes, experience with integrating automated tests into CI/CD pipelines, and the ability to map tests to formal requirements for traceability. Focus on those who can demonstrate analytical thinking for non-deterministic AI systems and have collaborated closely with both Data Scientists and Systems Engineers. Avoid candidates whose experience is limited to basic UI or manual testing, or who lack exposure to data-driven or AI/ML systems. The most important factor is proven expertise in building robust, automated validation frameworks for complex, probabilistic systems where reliability and compliance are critical.

What's IMPORTANT for this job?

-

Client and Project information (can be shared confidentially)

Budget: 40 EUR / h max