Case study

A Second Opinion by Design: Dual-Model Validation for Financial Document Extraction

Outcome

94.7% extraction accuracy

Client

Fintech company, financial document processing (name withheld)

Industry

Fintech

Status

In production

My role

Architecture and implementation lead

Published

September 1, 2026

The client is not named under contractual confidentiality. Where this write-up says "we", it refers to the delivery team I led; the architecture decisions described are mine.

Context

The client processes high volumes of financial documents, invoices, bank statements, and regulatory filings, and needed to extract structured data from them reliably enough to feed directly into downstream financial systems. Extracting structured fields from unstructured documents with a single AI model sounds like an efficient design on paper. The trouble is that a single-model extraction step can be confidently and silently wrong, and in a financial context, nobody notices a wrong field until it causes a discrepancy somewhere downstream, at which point it is far more expensive to trace back and fix than it would have been to catch at the source.

The Problem

The technical framing here is really a trust problem more than an extraction-accuracy problem. Financial documents are heterogeneous: an invoice from one vendor looks nothing like an invoice from another, bank statement formats vary by institution, and regulatory filings carry their own structure entirely. A model that performs well on the specific documents you happened to evaluate it on can still fail quietly on a format it has not seen, and there was no existing mechanism to catch that kind of failure before the extracted data moved downstream.

The other gap was measurement. A one-time benchmark test tells you how a model performed against the documents you had on the day you ran it. It does not tell you anything about how that same pipeline is performing six months later against real-world documents that have drifted in format, quality, and edge cases since then. Without a way to continuously assess extraction quality on live data, degradation would only surface once someone downstream noticed something did not add up.

Architecture

We built the pipeline around ingestion with LlamaParser, followed by a dual-model architecture that functions as a built-in second opinion rather than a single point of failure. One model, Claude, performs the actual extraction of structured fields from the parsed document. A second, independent model, based on Gemini, then reviews that extraction specifically for accuracy, completeness, and conformance to the required schema. Nothing is passed downstream to the client’s systems until it clears this internal quality gate.

The choice to use a genuinely different model family for the review step, rather than a second call to the same model, was deliberate. Two calls to the same underlying model are correlated: if that model has a systematic blind spot on a particular document format or field type, asking it to check its own work a second time will not reliably surface that blind spot. An independent model is more likely to catch a mistake the first one made, because its failure modes are not the same failure modes.

On top of the extraction and review pipeline, we built custom evaluation metrics that continuously and automatically assess data quality on every single extraction that runs, rather than relying on a benchmark run once at build time. That gave the client an ongoing view of extraction quality as their real document mix evolved, instead of a snapshot that goes stale the day after it is measured.

Results

The system achieves 94.7 percent accuracy across invoices, bank statements, and regulatory documents, and reduced manual processing time by 78.3 percent. The dual-model validation pattern proved effective enough that it became the default architecture for every AI feature built afterward in that environment, not just this one pipeline. That is the outcome I would flag to another engineering leader considering this approach: the pattern’s value showed up not just in this system’s numbers, but in how much it changed the default way the team approached every subsequent AI feature that touched anything financially sensitive.

What I Would Do Differently

The honest limitation of this design is that dual-model validation catches disagreement between the two models, but it does not catch a case where both models happen to be wrong in the same way, for instance a document field that is genuinely ambiguous or a formatting convention that is unusual enough to confuse any general-purpose model regardless of vendor. I would add a thin layer of deterministic, rule-based checks for the handful of fields where correctness matters most (totals reconciling, dates falling in valid ranges, and so on), specifically to catch the correlated-error case that two LLMs agreeing with each other cannot rule out on its own. We talked about this during the build and deprioritized it in favor of shipping the core dual-model gate first, which was the right call for time to value, but it is the next thing I would add rather than something I would skip indefinitely.