An LLM answer is a sample, not an average
Interactive demo of how large language models pick the next token: softmax probabilities, temperature, top-k and nucleus (top-p) sampling, repeated asks, and majority-vote self-consistency, framed for claims triage, financial summaries and number extraction.
A large language model does not look up an answer. At each step it produces a probability distribution over possible next tokens and then draws one. Every response is therefore a sample from a distribution, which is why the same prompt can return different answers, and why the most fluent answer is not necessarily the right one. This module makes that distribution visible for three tasks a finance or insurance team might hand to an LLM: labelling a claim note, summarising quarterly results, and extracting a number from a document.
Temperature rescales the model's scores before the softmax: low temperature concentrates probability on the top token, high temperature flattens the distribution. Top-k keeps only the k most likely tokens; top-p (nucleus sampling) keeps the smallest set whose combined probability reaches p. The bar chart shows the base distribution against the final sampling distribution after all three, with entropy and the effective number of choices. There is no live model on the page: the token probabilities are illustrative, hand-set values for a small vocabulary, but the sampling mechanics are exactly those applied to real model logits.
Asking the same question many times reveals the spread a single answer hides. Taking a majority vote across several calls (self-consistency) raises accuracy when the model's most likely answer is right: in the claims-triage example, from 52% with one call to 75% with five and 97% with 25. It locks in the error when the most likely answer is wrong: in the earnings-summary example the top word is "strong" although the source supports "modest", and voting barely helps (21% with one call, 26% with 25) and, with enough calls, settles on the wrong word. Greedy decoding returns the mode of the distribution, not the truth. For decisions, that argues for low temperature, repeated sampling or calibrated probabilities, and validation against labelled cases, with prompts, settings and outputs logged for model risk management.
Softmax with temperature
p_i = exp(z_i / T) / Σ_j exp(z_j / T)
z_i is the model's score (logit) for candidate token i, T the temperature and p_i the resulting probability. T → 0 gives greedy (argmax) decoding, T = 1 leaves the distribution unchanged, and large T approaches a uniform distribution.
Frequently asked questions
What does temperature do in an LLM?
It divides the model's logits before the softmax. Temperatures below 1 sharpen the distribution toward the most likely token and make outputs more repeatable; temperatures above 1 flatten it and make outputs more varied. It changes how the model samples, not what it knows.
What is top-p (nucleus) sampling?
A truncation rule that keeps only the smallest set of most-likely tokens whose combined probability reaches p, renormalises, and samples from that set. It cuts the long tail of unlikely tokens while adapting to how confident the model is at each step, unlike a fixed top-k.
Does setting temperature to zero make an LLM correct?
No. Temperature zero returns the single most likely token, the mode of the model's distribution. If that distribution favours a wrong answer, greedy decoding returns the wrong answer every time, with no variation to warn you. It improves repeatability, not accuracy.
Interactive explorer
The live model with sliders, charts and real-time statistics loads on this page. If it does not appear, enable JavaScript or open https://www.beyondaverages.org/llm-sampling in a modern browser.
Related modules
- Dice & convolution: Interactive 3D dice simulator. Roll up to five dice and watch the empirical sum histogram converge to the exact convolution probabilities: uniform to triangular to bell-shaped, the central limit theorem made visible.
- Cards & hypergeometric: Interactive card-dealing simulator. Deal hands from a 3D deck and compare empirical frequencies of aces, pairs and colors against exact hypergeometric probabilities, and see why a deck's memory matters.
- Frequency: Interactive Poisson distribution explorer for insurance claim counts and hospital admissions. Move the rate λ and watch P(X=k), the mode, and the probability of zero or extreme counts update live.
- Severity: Interactive loss severity explorer comparing Log-Normal, Pareto (heavy-tail) and Gamma distributions. Adjust parameters and see the mean, median, 95th and 99th percentile claim sizes, and the share of total cost from the largest 5% of claims, update in real time.