Middle QA Automation Engineer ( Python)

Description

We are building an AI powered document search assistant for a large enterprise client in Japan. Project design documentation accumulated over roughly twenty years currently sits on an isolated file server with no search capability at all, which forces engineers to rely on subject matter experts to locate specifications manually. The solution migrates this content into SharePoint within the client internal network and delivers natural language search across it. Roughly half of the corpus is in legacy Excel format where critical information lives inside shapes, callouts and embedded screenshots rather than in cells, so the pipeline combines format conversion, drawing layer parsing and OCR before indexing. The architecture is built on Azure (Python-based processing, Azure AI Search, Azure AI Document Intelligence) with Google Cloud considered as an alternative track. The current stage is a Proof of Concept focused on two highest-risk areas, unstructured data extraction and incremental index updates, with production rollout to follow.

Requirements

• Possessing 4+ years of dedicated professional experience in quality assurance and automation engineering, demonstrating a strong track record of successful project contributions.
• Demonstrated advanced proficiency in Quality Assurance methodologies and practices, ensuring high standards of software reliability and performance.
• Possessing intermediate hands-on experience in QA Automation, including the design, development, and maintenance of automated test suites.
• Strong working knowledge and intermediate-level experience with Python programming for scripting, test automation, and data analysis.
• A foundational understanding and beginner-level exposure to Artificial Intelligence concepts, with an eagerness to explore its applications in QA.

Preferred Skills:

• Familiarity with Microsoft Azure cloud services at a beginner level, particularly concerning its potential use in test environments or automation pipelines.

Job responsibilities

• Define and own the evaluation framework for the PoC, including extraction accuracy and search relevance metrics
• Validate extraction of unstructured content from legacy Excel files, covering text inside shapes, callouts, and embedded images processed through OCR
• Verify that content survives format conversion without loss, comparing source documents against indexed output
• Test incremental index update logic, confirming that only changed and new files are reprocessed and that deletions propagate correctly
• Measure and report index update latency and delta correctness
• Design test cases for edge scenarios (corrupted legacy files, macro-heavy workbooks, scanned images with low quality, near duplicate and outdated documents)
• Build a ground truth dataset together with the client subject matter experts and use it to score natural language search results
• Perform integration and regression testing across the ingestion, extraction, and indexing pipeline
• Support client validation sessions and prepare evaluation results for stakeholder review