Conformal prediction: knowing what you don't know
Interactive conformal prediction demo: wrap any model in prediction intervals or prediction sets with a distribution-free coverage guarantee. Compare constant and adaptive bands for claim costs, use 'not sure' sets to route AI triage to a human, and see what breaks under regime change.
Most models, from a regression line to a large language model, return a single best guess and say nothing about how wrong it might be. Conformal prediction is a simple, model-agnostic way to fix that. Hold back a calibration set, measure how badly the model misses on it, and use the right order statistic of those misses to draw an interval (for numbers) or a set of labels (for categories) around every new prediction. If calibration and future data are exchangeable, the result covers the truth with at least the probability you ask for, on average across cases, in finite samples and with no assumptions about the distribution.
The interval view predicts claim cost from vehicle value on synthetic data where the spread grows with value. At a 90% target, a constant-width conformal band hits 90% overall but covers almost every cheap vehicle and only about 72% of the most expensive ones; a normalised, adaptive band also hits 90%, stays close to 90% across the value range, and is narrower on average. A calibration-size slider shows the finite-sample guarantee at work, and a regime-change toggle (10% cost inflation and 50% higher volatility in the test period) drops coverage to about 78%, because the guarantee assumes the future resembles the calibration data.
The prediction-set view applies the same idea to a classifier triaging claims into five lines of business. Confident cases get a single label and can be automated; uncertain ones get two or more labels, a natural signal to route them to a human. At the default 90% target, a model whose top choice is wrong 21% of the time automates 58% of claims with under 5% error among those automated, and refers the rest. The data are synthetic; the method wraps any model, including an LLM classifier.
Split conformal quantile and coverage guarantee
q̂ = the ⌈(n + 1)(1 − α)⌉-th smallest calibration score, C(x) = { y : s(x, y) ≤ q̂ }, P(Y ∈ C(X)) ≥ 1 − α
n is the number of calibration cases and 1 − α the target coverage. s(x, y) is a non-conformity score measuring how unusual outcome y is for input x, such as |y − ŷ(x)| for regression or 1 − p̂(y | x) for classification; C(x) is the interval or label set. The guarantee is marginal (averaged over cases) and requires exchangeable data.
Frequently asked questions
What is conformal prediction?
A method for turning any model's point predictions into prediction intervals or sets with a guaranteed coverage rate. It calibrates the size of the interval on held-out data using the empirical distribution of the model's errors, so it needs no assumptions about the model or the data distribution beyond exchangeability.
Does conformal prediction guarantee coverage for every individual prediction?
No. The guarantee is marginal, averaged over all future cases. A constant-width interval can over-cover easy cases and under-cover hard ones while still hitting the target overall. Adaptive scores, such as residuals scaled by an estimate of local spread, bring coverage for each kind of case closer to the target.
When does conformal prediction fail?
When future data are not exchangeable with the calibration data: a regime change, inflation, a new customer mix or model drift. The coverage guarantee then no longer holds, so production systems should monitor realised coverage and recalibrate when it drifts.
Interactive explorer
The live model with sliders, charts and real-time statistics loads on this page. If it does not appear, enable JavaScript or open https://www.beyondaverages.org/conformal in a modern browser.
Related modules
- Dice & convolution: Interactive 3D dice simulator. Roll up to five dice and watch the empirical sum histogram converge to the exact convolution probabilities: uniform to triangular to bell-shaped, the central limit theorem made visible.
- Cards & hypergeometric: Interactive card-dealing simulator. Deal hands from a 3D deck and compare empirical frequencies of aces, pairs and colors against exact hypergeometric probabilities, and see why a deck's memory matters.
- Frequency: Interactive Poisson distribution explorer for insurance claim counts and hospital admissions. Move the rate λ and watch P(X=k), the mode, and the probability of zero or extreme counts update live.
- Severity: Interactive loss severity explorer comparing Log-Normal, Pareto (heavy-tail) and Gamma distributions. Adjust parameters and see the mean, median, 95th and 99th percentile claim sizes, and the share of total cost from the largest 5% of claims, update in real time.