Senior Automation QA Engineer – Python and LLM Testing

We are looking for a Automation QA Engineer to join our team in Poland. The role will primarily focus on API testing (80%) with UI testing (20%), alongside testing LLM-powered AI solutions. You will help design, enhance, and maintain automation frameworks, ensuring efficient and scalable testing—especially for AI/LLM-driven features.

Key Responsibilities:

Automation Framework & Strategy

• Contribute to designing and improving automation testing strategies.
• Develop and maintain automation frameworks for API, UI, and LLM validation.
• Ensure scalability, maintainability, and reusability of automation code.

Test Development & Execution

• API Testing: Develop automated API tests in Python using the Requests library.
• UI Testing: Create and maintain UI tests with Playwright.
• LLM Testing: Design and execute automated test suites for Large Language Model (LLM)based features, including prompt validation, output accuracy, and bias/error detection.
• Integrate automated tests with Postman collections where applicable.
• Execute performance, regression, and functional tests.

Continuous Integration & Deployment

• Integrate automation tests into CI/CD pipelines (Jenkins preferred).
• Work with Docker and Kubernetes to manage and scale test environments.

Collaboration & Documentation

• Collaborate with developers, QA, and DevOps teams to improve the QA process.
• Write test plans, strategies, and design documents.
• Track issues using Jira and manage test cases (Xray optional).
• Analyze logs, investigate failures, and troubleshoot in staging and production.

Required Qualifications & Experience:

✅ Must-Have:

• 5 years of experience in test automation or a related QA role.
• Strong programming skills in Python.
• Solid experience in API automation (Requests library) and UI automation (Playwright).
• Hands-on experience testing LLM-based applications (prompt engineering, output evaluation, hallucination/bias testing).
• Experience with CI/CD tools (Jenkins preferred).
• Experience with Docker & Kubernetes for test environments.
• Strong analytical, problem-solving, and debugging skills.
• Experience with Jira for tracking and reporting.
• Ability to work independently and proactively in a fast-paced environment.

✅ Nice to Have:

• Experience with Kafka.
• Familiarity with Xray for test management.
• Knowledge of AWS cloud services.
• Experience with ML/AI model evaluation metrics.