Fraud detection: decision trees, random forests and the investigation queue
Interactive fraud detection demo on 1,600 synthetic insurance claims: compare logistic regression, a CART decision tree and a random forest by their decision boundaries, watch deeper trees overfit, and judge models by precision and recall at your investigators' capacity.
Claims fraud is a needle-in-a-haystack problem: a few percent of claims, hidden among honest ones, and an investigations unit that can review only so many. This module builds that problem on 1,600 synthetic motor claims described by two features (days from policy start to claim, and claim size relative to the average). Fraud concentrates in large claims made soon after a policy starts or just before it expires, while large mid-term claims are usually genuine, so the obvious linear rule, bigger is riskier, is the wrong one.
Three models are fitted live and drawn as shaded probability maps over the data. Logistic regression can only draw a straight boundary and misses both pockets. A CART decision tree carves the plane into rectangles and finds them, and a depth slider shows it memorising noise as it grows, with training and test performance pulling apart. A random forest averages thirty bootstrapped trees to keep most of the flexibility while smoothing away much of the overfitting.
Evaluation is the practical heart. With 5% fraud (the default), a model that never refers anyone is 95% accurate, so accuracy is useless. ROC-AUC can mislead too: on this book logistic regression has the highest test AUC of the three (0.91) but by far the lowest average precision (0.31, against 0.62 for the tree and 0.73 for the forest). The useful questions are the ones an investigations manager asks: if we review the top K% of claims, what share are fraudulent (precision@K) and what share of all fraud do we catch (recall@K)? Reviewing the top 5% of test claims, the tree and the forest each find 17 of the 24 frauds and the straight line finds 9. The data are synthetic and the models illustrative.
Gini impurity and the CART split rule
G(node) = 1 − Σ_c p_c², split maximises G(parent) − (n_L/n)·G(L) − (n_R/n)·G(R)
p_c is the share of class c (fraud or legitimate) in a node; n, n_L and n_R are the numbers of claims in the parent and in the left and right children. Each split is the single feature-threshold pair that most reduces weighted impurity; a random forest averages many such trees grown on bootstrap samples.
Frequently asked questions
Why is accuracy a bad metric for fraud detection?
Because fraud is rare. If 5% of claims are fraudulent, labelling every claim legitimate scores 95% accuracy while catching nothing. Precision (how many referrals are real fraud), recall (how much fraud is caught) and average precision describe performance on the class that matters.
Why do decision trees beat logistic regression on some problems?
Trees capture non-linear effects and interactions automatically (such as 'large claim AND early in the policy') by splitting the feature space into regions. Logistic regression needs those effects engineered by hand. The cost is variance: deep trees overfit, which is what pruning, minimum leaf sizes and ensembles control.
How should a fraud model be evaluated against investigation capacity?
Fix the number of claims investigators can review, rank claims by model score, and measure precision@K and recall@K at that capacity. Two models with similar ROC-AUC can differ sharply in how much fraud they surface in the top few percent, which is the only part of the ranking that gets acted on.
Interactive explorer
The live model with sliders, charts and real-time statistics loads on this page. If it does not appear, enable JavaScript or open https://www.beyondaverages.org/fraud-detection in a modern browser.
Related modules
- Dice & convolution: Interactive 3D dice simulator. Roll up to five dice and watch the empirical sum histogram converge to the exact convolution probabilities: uniform to triangular to bell-shaped, the central limit theorem made visible.
- Cards & hypergeometric: Interactive card-dealing simulator. Deal hands from a 3D deck and compare empirical frequencies of aces, pairs and colors against exact hypergeometric probabilities, and see why a deck's memory matters.
- Frequency: Interactive Poisson distribution explorer for insurance claim counts and hospital admissions. Move the rate λ and watch P(X=k), the mode, and the probability of zero or extreme counts update live.
- Severity: Interactive loss severity explorer comparing Log-Normal, Pareto (heavy-tail) and Gamma distributions. Adjust parameters and see the mean, median, 95th and 99th percentile claim sizes, and the share of total cost from the largest 5% of claims, update in real time.