Your Fraud Model's Scores Are Not Probabilities

A fraud model trained on last year's data ships with a score threshold of 0.65. Six months later, fraud rates shift. A product manager asks: "at score 0.8, what's our expected false positive rate?" An ops team wants to build a cost model: "if we automatically block scores above 0.75, what does that cost us in declined good transactions?" A downstream system weights its decision by 1 - fraud_score.

All of these questions assume the model's output is a probability. Almost certainly, it isn't.

Why raw scores aren't probabilities

Most classifiers — gradient boosted trees, neural nets, logistic regression with regularization — are optimized for ranking, not probability estimation. The training objective pushes fraud cases above non-fraud cases. It does not push the output distribution to match observed event rates.

The result: a model can have near-perfect AUC while being systematically wrong about the magnitude of its scores. A score of 0.8 might correspond to 40% empirical fraud rate. Or 2%. The model doesn't know. Neither do you, until you measure it.

A quick check: the reliability diagram

Bin your held-out predictions into deciles. For each bin, plot mean predicted score against mean actual outcome rate. A perfectly calibrated model produces points along the diagonal. Most production models look like this instead:

Actual
fraud    1.0 ┤                                              ●
rate         │                                         ●
         0.8 ┤                                    ●
             │         ····················
         0.6 ┤              ·           ●
             │          ·
         0.4 ┤       ·          ●
             │     ·
         0.2 ┤   ·    ●    ●
             │  ·  ●
         0.0 ┼──────────────────────────────────────────────
             0.0  0.2  0.4  0.6  0.8  1.0   Predicted score

····  = perfect calibration (diagonal)
●     = model's actual bins

The S-curve shape is typical of gradient boosted trees: overconfident in the middle, underconfident at the extremes. Logistic regression tends to go the other way. The exact shape depends on training data balance, regularization strength, and feature distributions — not on whether your model is "good."

What miscalibration actually costs

In isolation, miscalibration is invisible. AUC doesn't catch it. Precision and recall at a single threshold don't catch it. You only see it when you:

• Use scores as inputs to a downstream system — a rules engine, a cost model, a human review queue prioritizer — that treats them as probabilities
• Compare thresholds across cohorts — score 0.7 in the US vs. score 0.7 in a new market where base rates differ
• Reuse a threshold after a data shift — the threshold was set when fraud rate was 1.2%; now it's 0.6%; the same threshold now behaves differently even if the model's ranking is unchanged
• Run expected-value calculations — "block if P(fraud) × transaction_value > review_cost" — where P(fraud) is read directly from the model output

The last one is especially common. It's also where the error compounds: a 2× miscalibration in the score translates to a 2× error in expected loss, which breaks the decision boundary entirely.

Two calibration methods and when to use each

Platt scaling

Fit a logistic regression on a held-out calibration set, using the model's raw score as the only feature:

from sklearn.linear_model import LogisticRegression

cal = LogisticRegression()
cal.fit(val_scores.reshape(-1, 1), val_labels)
calibrated_probs = cal.predict_proba(test_scores.reshape(-1, 1))[:, 1]

The output is guaranteed to be in [0, 1] and is monotone in the input. The assumption: the miscalibration is sigmoid-shaped — symmetric around some midpoint. This is usually approximately true for scores that come from a single well-trained model.

Use when: you have limited calibration data (a few thousand examples), the model is a single estimator, and you want a simple, auditable fix.

Isotonic regression

Fit a non-decreasing step function instead of a sigmoid:

from sklearn.isotonic import IsotonicRegression

cal = IsotonicRegression(out_of_bounds="clip")
cal.fit(val_scores, val_labels)
calibrated_probs = cal.predict(test_scores)

No parametric assumption. Can fit any monotone relationship between score and outcome rate. The cost: it has more degrees of freedom and will overfit on small calibration sets. Cross-validate on a held-out split — not the training data, not the eval set you measured AUC on.

Use when: you have a large calibration set (tens of thousands), the model's miscalibration is nonlinear or asymmetric, or you're calibrating an ensemble whose score distribution is irregular.

The result

AUC is unchanged — calibration is a monotone transform, so it cannot affect ranking. But Brier score drops by 37% and ECE drops by 80%. The model now means what it says.

The part people skip: measuring calibration in production

Calibration degrades silently. The model's ranking quality — AUC — can stay flat while the calibration drifts, because ranking only requires the order to be preserved. A score shift of +0.1 across the board doesn't change a single AUC calculation. It does change every threshold decision and every downstream expected-value calculation.

The right monitoring setup: track ECE on a rolling window of labeled outcomes. Outcomes arrive with delay in fraud (chargebacks take 30–90 days), so you need a lagged monitoring pipeline. When ECE rises above some threshold, trigger recalibration — not retraining. Recalibration is cheap. You're fitting two parameters (Platt) or a step function (isotonic) on recent data. You don't need to retrain the base model.

Cohort calibration: the harder problem

A model calibrated on the overall population may be miscalibrated on subpopulations. A fraud model trained on US transactions may have well-calibrated scores in aggregate while systematically underscoring fraud in a new geography where the feature distributions differ. The base model is not wrong — its ranking may still be good — but the implied probabilities are wrong for that cohort.

The fix is the same: fit a separate calibrator per cohort on recent labeled data from that cohort. This requires enough labels per cohort, which is the real constraint. If a cohort is too small for reliable calibration, fall back to the global calibrator and flag the uncertainty.

Implementation

The calibration code behind this post is in calibrate-ml: Platt scaling, isotonic regression, ECE computation, and reliability diagram generation. The companion eval repo, ml-eval-report, handles Brier score, log loss, and the full threshold sweep.

The most important thing: run this on your current production model before doing anything else. The reliability diagram takes ten lines of code and a held-out labeled dataset. If the points are on the diagonal, you're fine. If they're not — and they usually aren't — you now know what you're dealing with.