AI Optimization Engineer (AI QA)

Who We Are:
Respaid is a Y Combinator-backed startup, revolutionizing modern B2B collections. We count Twilio, DoorDash (the US competitor of Uber Eats), and Redis among our clients. We move fast, iterate relentlessly, and leverage AI to drive revenue growth at scale. If you’re obsessed with scraping, automation, and cutting-edge AI, this role is for you. 

What You’ll Do:
 

AI & QA Responsibilities

• Dive into our existing AI agents (voice and text) tools, and models to evaluate and optimize their behavior, accuracy, and robustness.
• Design and execute AI-specific test scenarios, especially around STT and TTS responses, recommendation systems, ranking algorithms with metrics, etc.
• Identify, document, and help fix AI-specific bugs and inconsistencies, including hallucinations, poor ranking logic, or word-sensitive behavior.
• Collaborate closely with ML engineers and product teams to translate test findings into better model behavior and improved user experience.
• Suggest improvements or simplifications to boost both accuracy and stability of model behaviors.
• Provide feedback based on test data and user feedback to drive continuous optimizations and refinement.
• Participate in processes: weekly planning, retrospectives, demos, and daily standups.

Model Optimization

• Assess and implement opportunities for performance, latency, and accuracy improvements in AI agents.
• Contribute to model tuning and reconfiguration based on observed output quality or business priorities.
• Help integrate LLM as a judhe model testing and validation processes within the ML lifecycle.

Performance Monitoring and Maintenance

• Set up and monitor key metrics to track AI agents behavior.
• Create and use tools (Langfuse, Vocera etc) to investigate edge cases, failures, or degradations.

Tech Stack & Tools:

• Python
• REST API Testing (Postman, Swagger, DevTools)
• Logging and Monitoring
• Databases ( MySQL)
• Test Case Management (TestRail)
• Project Management (Jira, Confluence)

What We’re Looking For:

• Strong QA or dev engineering background with a deep understanding of testing methodologies and processes.
• Hands-on experience with AI systems and a drive to improve them.
• Ability to diagnose complex AI bugs and edge-case failures.
• Familiarity with machine learning concepts, models, and optimization techniques.
• Strong communication skills and English (we are international team)

Nice to Have:

• Scripting skills for test automation or analysis (Python, etc.)
• Experience with CI/CD pipelines for model deployment/testing
• Exposure to A/B testing and evaluation of ML model changes
• Laravel, PHP.

Are you passionate about AI and quality? Do you have a sharp eye for detail and a deep curiosity for how intelligent systems work under the hood? We’re looking for an AI QA & Model optimization Engineer to help us polish and elevate our AI tools and agents to production-grade performance.