Senior AI Quality Tester

About the role
You’ll validate AI-driven systems (ML models, LLM solutions, and Agentic AI workflows) to ensure outputs are accurate, reliable, secure, and compliant with governance standards. Focus areas include model behavior, prompt reliability, automation flows, and enterprise integration testing.

What you’ll do

• Test ML models using metrics (accuracy, precision, recall, F1) and adversarial/edge-case testing 
• Validate LLM outputs for correctness, relevance, and hallucination control 
• Run prompt-variant testing for response consistency; detect bias/incorrect reasoning; enforce business rules 
• Test multi-agent workflows, orchestration, decision logic, fallback handling, automation scenarios 
• Perform end-to-end scenario testing across APIs/services; validate real-time inference pipelines 
• Governance/compliance testing: RBAC/data access controls, sensitive data handling, auditability/traceability 
• Performance/reliability: load/latency/scalability testing; monitor drift/degradation over time 

Must-have

• 5+ years QA/testing with exposure to AI systems; experience testing ML models, APIs, enterprise apps 
• LLM evaluation techniques (hallucination detection, prompt testing) + dataset validation/test data generation 
• Automation tools: Selenium/Cypress/Playwright; API testing (Postman/REST Assured) 
• Automation scripting (Python or JavaScript); basic Python for AI pipeline testing 
• Azure environment awareness; CI/CD pipelines (Azure DevOps); logging/monitoring awareness 
• Familiarity with embeddings/vector DBs and ML/LLM workflow concepts 

Authority / constraints

• Owns AI testing standards/metrics and acceptance criteria; cannot approve release without quality thresholds 

Nice-to-have certs

• ISTQB Advanced Test Analyst preferred; AI-900 recommended