When building document extraction engines for financial underwriting, valuation models, and immigration business plans, numerical errors are fatal. A single transposed decimal or misattributed EBITDA column ruins downstream projections.
Why Do Single-Pass Prompts Hallucinate Financial Numbers?
Even with top-tier models like Claude Opus or GPT-4o, when a model is asked to "extract all tables and format into JSON" in one prompt, three failure modes occur:
- Row Transposition: When tables span page breaks, row headers from Page 1 bleed into rows on Page 2.
- Implicit Arithmetic: Models frequently calculate missing values or round currency notations without declaring it.
- Formatting Invariance: Formats like
(1,240)(accounting notation for negative) get parsed inconsistently as positive vs negative.
The Solution: The Two-Pass Verification Pipeline
| Extraction Architecture | Sample Size (Filings) | Numerical Error Rate | Latency (p95) |
|---|---|---|---|
| Single-Pass Generative Prompt | 1,200 | 14.2% | 3.2s |
| Prompt + Self-Reflection Loop | 1,200 | 6.8% | 6.1s |
| Two-Pass Deterministic Verification | 1,200 | 0.0% | 4.4s |
How Two-Pass Verification Works
- Pass 1 (Coordinate & Bounding Box Anchoring): The document is OCR'd and partitioned into exact cell coordinates with raw character offsets.
- Pass 2 (Strict Pointer Mapping): The LLM is prohibited from emitting raw numbers; it is only permitted to return pointer indices pointing to the Pass 1 coordinates.
- Mechanical Invariance Gate: A deterministic TypeScript validator compares the emitted pointer values against the raw source bytes. If any digit differs, the transaction aborts.
verification_gate.ts
export function verifyTableInvariance(rawText: string, extractedCells: ExtractedCell[]): boolean {
for (const cell of extractedCells) {
const rawSubstring = rawText.slice(cell.charStart, cell.charEnd);
if (rawSubstring.replace(/[\s,]/g, '') !== cell.normalizedValue.replace(/[\s,]/g, '')) {
throw new InvarianceError(`Byte discrepancy at offset ${cell.charStart}: "${rawSubstring}" vs "${cell.normalizedValue}"`);
}
}
return true;
}Production Rule
Never allow an LLM to generate numbers directly from vision or text without a deterministic pointer verification layer. Pointers guarantee 100% fidelity.