Engineering & Research

Notes from Production

Empirical measurements, benchmark results, and hard-learned reliability patterns from operating live LLM agents, RAG systems, and AI workflows.

All Notes LLM Evals & Benchmarks Agent Reliability System Architecture
LLM Evals & Benchmarks

Why Your LLM Judge Grades Its Own Homework: Measuring 23.6% Preference Leakage in Automated Evals

When automated evaluations use the same model family for generation and scoring, measured scores inflate dramatically. Here is empirical data across 140 test runs and how multi-model grading solves it.

Key Measured Delta: Same-model generator/judge pairings inflated evaluation scores by +23.6% on subjective safety rubrics compared to independent cross-model grading.
6 min read · By Uriel Naor Read Article →
Agent Reliability

Zero Hallucinations in Multi-Table Extraction: The Strict Two-Pass Verification Pattern

Extracting dense financial tables from scanned PDFs frequently causes silent hallucinations. We implemented a mechanical byte-invariance verification gate that reduced extraction errors from 14.2% to 0.0%.

Production Outcome: Eliminating unanchored generative passes cut field extraction failures from 14.2% to 0.0% across 1,200 complex multi-page financial filings.
5 min read · By Uriel Naor Read Article →
LLM Evals & Benchmarks

How Many Eval Seeds Do You Actually Need? Clopper-Pearson Bounds for 95% LLM Recall Claims

Claiming "95% recall" on an LLM agent with 20 test cases is mathematically meaningless. A reference lookup table for independent seed sizes, miss tolerances, and paired McNemar designs.

Key Math Finding: Proving ≥95% recall with a 95% Clopper-Pearson lower bound requires a minimum of 60 independent test seeds with 0 misses or 95 seeds with 1 miss.
8 min read · By Uriel Naor Read Article →