Senior SDET (LLM/AI)

A US product company - an athletics management platform used by thousands of schools across the United States. The team is adding AI features to the product, and this role owns the layer that proves they actually work.

This is an engineering role with a quality focus, not a pure QA position: roughly 70% evals and test infrastructure, 30% product features in the same stack as the rest of the team.

What you'll do

- Build and own eval harnesses, golden datasets and scoring for production agentic workflows
- Gate prompt, model and orchestration changes in CI - nothing merges without an eval verdict
- Own the Playwright web and Maestro mobile E2E suites
- Instrument AI features in production and turn real failures into new eval cases
- Ship user-facing features in Node, TypeScript, React and React Native - from spec to rollout

Time zone

The team is distributed across US time zones, with its core on US Central Time. We ask for a few core overlap hours daily - in practice from around 16:00–17:00 Kyiv time. 

Requirements

- 5+ years shipping production software with TypeScript, Node and React / React Native
- Ready to write product code, not only test and tooling code
- Hands-on experience with LLM/AI features in production: agentic workflows, prompt and model integration, LLM failure modes
- Playwright and CI/CD in GitHub Actions
- AWS (ECS/Lambda, Secrets Manager, CloudWatch)
- Daily use of frontier models and agentic coding tools

- Building LLM/agent evals: eval harnesses, golden datasets, LLM-as-a-judge scoring
- English B2+ -strong written async communication, direct work with the CTO

Nice to have
- Eval platforms: LangSmith, Braintrust, Langfuse
- MCP servers, agent SDKs, Maestro, domain experience in scheduling or logistics

We offer

- Funded frontier models from Anthropic and OpenAI, agentic coding tools
- Greenfield: the eval layer is yours to design from scratch
- Small team - weekly contact with the CTO, your work in production the week you ship it
- Fully remote