Model Calibration Lab
A record of picks tells you what won. Calibration tells you whether the probabilities meant anything: when we say 60%, does it happen 60% of the time? Below, every settled prediction, scored two ways — the sharp market (devigged) and our own model — with the uncomfortable comparison in plain sight.
Our model (Elo + form): 0.615
Our model (Elo + form): 1.025
Our model (Elo + form): 0.6pp
Predicted probability vs observed frequency
Each point is a 10-point probability band (min. 30 observations). The closer to the diagonal, the better calibrated. Every match contributes its three outcomes.
| Band | Sharp market (devigged) | Our model (Elo + form) | ||||
|---|---|---|---|---|---|---|
| Predicted (avg) | Observed | Obs. | Predicted (avg) | Observed | Obs. | |
| 0–10% | 7.3% | 7% | 488 | 7.5% | 7.6% | 238 |
| 10–20% | 16.2% | 17.2% | 2901 | 16.4% | 16.1% | 1875 |
| 20–30% | 25.3% | 24.9% | 9249 | 26% | 25.6% | 10791 |
| 30–40% | 34.3% | 34.4% | 4671 | 34.7% | 35.4% | 4588 |
| 40–50% | 44.7% | 43.9% | 2976 | 44.7% | 44.3% | 3588 |
| 50–60% | 54.6% | 55.6% | 1907 | 54.4% | 55.5% | 1752 |
| 60–70% | 64.3% | 64.2% | 1006 | 63.9% | 62.9% | 672 |
| 70–80% | 74.6% | 77.7% | 448 | 74.2% | 77.7% | 220 |
| 80–90% | 83.7% | 84.5% | 129 | 83.6% | 86.3% | 73 |
Lower Brier and log-loss are better; calibration error is the observation-weighted average gap between predicted and observed. The honest headline: the devigged sharp market forecasts better than our model (lower Brier score and log-loss). On calibration error alone both are within sampling noise of perfect calibration; the market wins by separating likely from unlikely outcomes more sharply — which is precisely why this site prices everything against the market instead of selling you model predictions. Methodology: probabilities as displayed at logging time, no retro-fitting; outcomes from official results.
Full methodology →Over/under 2.5 and both teams to score
Pinnacle's closing price with the margin removed, against what happened, in bands of 10 points (bands with at least 30 matches).
Over 2.5 goals
| Market said | Happened | ± (95%) | Matches |
|---|---|---|---|
| 27% | 25% | ±10 | 72 |
| 36% | 42% | ±5 | 345 |
| 46% | 47% | ±3 | 838 |
| 55% | 56% | ±3 | 1336 |
| 64% | 63% | ±3 | 861 |
| 73% | 71% | ±6 | 231 |
3,710 matches, Brier score 0.2395. ± is how far “happened” can drift from what the market said by chance alone with that many matches: a gap inside it is noise.
Both teams to score
| Market said | Happened | ± (95%) | Matches |
|---|---|---|---|
| 47% | 50% | ±8 | 134 |
| 55% | 55% | ±4 | 545 |
| 64% | 67% | ±5 | 360 |
| 73% | 64% | ±13 | 44 |
1,088 matches, Brier score 0.2386. ± is how far “happened” can drift from what the market said by chance alone with that many matches: a gap inside it is noise.
Correct-score estimates vs what happened
Every score from 0–0 to 6–6 of every match, estimated from the closing prices (Pinnacle, margin removed) with the same model as the match pages, grouped by estimated probability. If the estimates are honest, scores estimated at about 10% happen about 10% of the time.
| Estimated | Predicted (avg) | Happened | Scores |
|---|---|---|---|
| 0–3% | 0.5% | 0.5% | 134,837 |
| 3–6% | 4.4% | 4.5% | 19,694 |
| 6–9% | 7.4% | 7.5% | 12,948 |
| 9–12% | 10.1% | 9.9% | 6,563 |
| 12–15% | 13.3% | 13.1% | 2,706 |
| 15–100% | 16.5% | 16.5% | 387 |
3,615 matches: the most likely score was estimated at 12.4% on average and happened in 12.5% of them.
