Senior QA Engineer
This role sits within a fast-growing technology company developing advanced AI agents for the
construction industry, focused on automating the analysis of building models, drawings, and
regulatory documents to enhance compliance, planning, and design validation. As part of a
senior engineering team, this position plays a critical role in ensuring the reliability and
trustworthiness of complex, non-deterministic AI systems that support key decision-making
processes in construction projects.
Responsibilities:
• Design and execute comprehensive tests for agentic AI systems, focusing on tool selection,
multi-step reasoning, and recovery from failures
• Evaluate agent outputs for grounding, citation accuracy, and evidence-backed claims
• Develop and maintain automated frameworks to assess generative AI outputs for hallucination,
consistency, and factual accuracy
• Build and run guardrail and adversarial tests to identify vulnerabilities such as prompt injection
and instruction manipulation
• Validate multi-turn conversations and ensure correct handling of conversational state across
sessions
• Implement and monitor continuous evaluation of agent performance in production
environments
• Investigate and triage quality incidents, tracing failures to specific models or datasets and
collaborating on fixes
• Create and maintain robust integration and API tests to ensure seamless backend and
user-facing workflows
• Enforce quality gates and produce verification evidence to support confident releases
• Map test cases to system requirements, ensuring traceability and comprehensive coverage
Skills:
- Expert Python programming for test automation and custom evaluation tooling
- Hands-on experience with LLM/agent evaluation frameworks (e.g., DeepEval, TruLens,
RAGAS, or custom evaluators)
- Proficiency in designing and implementing non-deterministic and generative AI test strategies
- Strong knowledge of integration, API, and UI automation tools (e.g., Selenium, Playwright,
Requests)
- Experience with performance and load testing tools (e.g., Locust, JMeter, k6)
- Proficiency with SQL and data validation tools (e.g., Great Expectations)
- Familiarity with vector databases and retrieval corpora
- Experience integrating automated tests into CI/CD pipelines (e.g., GitLab)
- Ability to define and communicate pass/fail criteria for probabilistic systems
- Strong collaboration and communication skills with engineering and product teams