Architecture Designer - LLM Extraction Pipeline

Context

We are building a complex agent that analyzes documents and produces defined data structures extracted from them. Three properties are critical: the structures must be accurate — faithful to the source documents, with exact references and exact quotes; valid — types conforming to their schemas, required fields filled, and so on; and free of duplicates — not repeating structures that were already accepted or rejected before.

Must-have

- 3+ shipped LLM-powered systems in production, each with a measurable quality harness the candidate built personally (golden sets, precision/recall tracking, regression gates in CI). Be ready to walk through one in depth — at whatever level your own NDAs allow — including a concrete regression the harness caught and what changed because of it.
- Proven ability to split model work from deterministic validation: pipelines where extraction is model-driven but every acceptance check (dedup, linkage, schema, rule-conformance) is executable code. Prompt-only quality control is a disqualifier.
- Information-retrieval and data-modeling depth: hybrid exact+semantic indexing; identity-keyed deduplication (record identity over embedding similarity); append-only stores with supersession semantics; incremental re-indexing without full rebuilds.
- Multi-pass agent-pipeline engineering: idempotent, sumable fan-out/verify loops (extract -> validate -> dedup/link -> adversarial verify -> loop-until-dry), token/cost budgeting, caching.
- Evidence-first design method: for every architectural element you propose, you state its precondition and how it is verified; you back your designs with executable acceptance checks against golden sets, not documents alone.

Nice-to-have

- Experience working with and analyzing regulated-domain documents (insurance/finance/legal).
- Event-sourced or lakehouse-style record stores (append-only, versioned, time-travel).
- Knowledge-graph construction and record linkage / entity resolution.
- Adversarial-verification agent patterns (a dedicated verifier stage that tries to refute extractions).
- Custom eval-suite tooling built when off-the-shelf eval frameworks did not fit.