AI engineering teams frequently publish evaluation numbers like "Our agent scored 96% accuracy on our benchmark suite!" without disclosing sample size or confidence intervals. In high-compliance environments (e.g. child safety monitoring, legal reasoning, automated financial underwriting), point estimates without statistical bounds are useless.
The Exact Clopper-Pearson Sample Size Table
To prove that your agent meets a true one-sided 95% lower bound for target recall, use this exact lookup table:
| Target Lower Bound Recall | Allowed Failures (k) | Required Independent Seeds (N) | Effective Sample Rule |
|---|---|---|---|
| ≥ 90% Recall | 0 misses | 30 seeds | Distinct scenarios |
| ≥ 95% Recall | 0 misses | 60 seeds | Zero tolerance |
| ≥ 95% Recall | 1 miss | 95 seeds | 1 production miss tolerated |
| ≥ 95% Recall | 2 misses | 126 seeds | 2 production misses tolerated |
| ≥ 98% Recall | 0 misses | 150 seeds | Critical safety bar |
The Paraphrasing Trap: Pseudoreplication
A widespread trap in LLM benchmarking is taking 5 prompt templates and generating 10 LLM paraphrases for each to report "50 test runs".
Because the semantic underlying distribution remains tightly clustered around the original 5 templates, the true degrees of freedom are 5. This causes catastrophic statistical overconfidence.
Paired Ablation Designs (McNemar's Exact Test)
When comparing a new agent feature (e.g., Memory vs No-Memory, Tool Use vs Direct Prompt), never use independent two-sample t-tests. Since both configurations run on the exact same seed cases, you must use a paired McNemar exact test or Wilcoxon signed-rank test on discordant pairs:
from statsmodels.stats.contingency_tables import mcnemar
# b: passed in v2 but failed in v1
# c: passed in v1 but failed in v2
table = [[both_pass, b_v2_only],
[c_v1_only, both_fail]]
result = mcnemar(table, exact=True)
print(f"p-value: {result.pvalue:.4f}")