Zero errors in 50 tests is not a zero error rate
Interactive guide to validating an AI or LLM tool's error rate: exact Clopper-Pearson and Wilson intervals, the rule of three, the number of test cases needed to rule out a target error rate, and the probability that a flawed model passes your validation.
When a firm deploys an AI tool (extracting fields from claim forms, summarising credit memos, classifying complaints), someone has to decide whether its error rate is low enough. The usual evidence is a test set: run the tool on n labelled cases and count the mistakes. This module shows what that evidence can and cannot support. Zero errors in 50 cases sounds reassuring, but the exact one-sided 95% upper bound on the true error rate is still about 5.8%, close to the 3/n = 6% given by the rule of three.
The interval view compares the exact Clopper–Pearson interval with the Wilson and normal-approximation (Wald) intervals for any number of tests and observed errors, and shows why the Wald interval collapses to a meaningless [0%, 0%] when no errors are seen. The planning view runs the calculation the other way: to show an error rate is below a target with given confidence, how many cases are needed if zero, one or a few errors are allowed, and what labelling them will cost. Ruling out a 1% error rate at 95% confidence with no failures takes 299 cases.
The simulation view asks the model-risk question directly: if the true error rate were 2%, how often would a 50-case validation that allows one error approve it? The exact answer is about 74% of the time, and 200 simulated campaigns show it happening. The practical lessons follow: size validation from the error rate you must rule out, stratify test cases across document types and edge cases, and re-validate when the model, vendor version or prompt changes.
Zero-failure sample size
n = ⌈ ln(α) / ln(1 − p₀) ⌉ ≈ 3 / p₀ at 95%
n is the number of error-free test cases needed to show, with confidence 1 − α, that the true error rate is below p₀ (α = 0.05 at 95% confidence, so ln(α) ≈ −3). More generally, the exact Clopper–Pearson upper bound p_U after x errors in n cases solves P(X ≤ x | n, p_U) = α.
Frequently asked questions
If an AI model makes no errors on 100 test cases, what is its error rate?
Unknown, but bounded: with zero failures in n trials the exact one-sided 95% upper bound is 1 − 0.05^(1/n), about 3/n. For 100 cases the true error rate could still be as high as about 3% at 95% confidence.
What is the rule of three?
A shortcut for the 95% upper confidence bound on a proportion when zero events are observed in n independent trials: approximately 3/n. It comes from solving (1 − p)^n = 0.05, and it is within about 5% of the exact bound once n is 30 or more.
Why not use the normal approximation for AI error-rate intervals?
Because error rates are small and samples are modest, exactly where the Wald interval p̂ ± z·√(p̂(1 − p̂)/n) fails: at zero errors it gives a zero-width interval. Exact Clopper–Pearson or Wilson intervals keep valid coverage and should be the default for validation reports.
Interactive explorer
The live model with sliders, charts and real-time statistics loads on this page. If it does not appear, enable JavaScript or open https://www.beyondaverages.org/ai-validation in a modern browser.
Related modules
- Dice & convolution: Interactive 3D dice simulator. Roll up to five dice and watch the empirical sum histogram converge to the exact convolution probabilities: uniform to triangular to bell-shaped, the central limit theorem made visible.
- Cards & hypergeometric: Interactive card-dealing simulator. Deal hands from a 3D deck and compare empirical frequencies of aces, pairs and colors against exact hypergeometric probabilities, and see why a deck's memory matters.
- Frequency: Interactive Poisson distribution explorer for insurance claim counts and hospital admissions. Move the rate λ and watch P(X=k), the mode, and the probability of zero or extreme counts update live.
- Severity: Interactive loss severity explorer comparing Log-Normal, Pareto (heavy-tail) and Gamma distributions. Adjust parameters and see the mean, median, 95th and 99th percentile claim sizes, and the share of total cost from the largest 5% of claims, update in real time.