Graded against reality.
Every probability we publish comes from a sensor we can score: when a prediction market resolves, reality hands us an answer key we did not write. This page is the referee’s ledger — the crowd consensus we cite, and the deterministic engine we run on top of it, graded with proper scoring rules. We grade to keep the probabilities we publish honest — not to outguess the crowd. Markets are a sensor we cite, never an opponent we score against; a tie with the crowd here is the system working, not an edge failing. As of 2026-07-13. Case-selection methodology →
The living benchmark
Forecasts are logged while each market is still open, then graded only after reality resolves it — the record cannot be back-filled, because the forecast exists before the answer does. So far 26,886 open-market forecasts have been captured and 13,472 have matured and been scored.
| forecaster | Brier (95% CI) | log loss | skill vs base rate | markets |
|---|---|---|---|---|
| Crowd consensus (the sensor we cite) | 0.0705 [0.0678, 0.0734] | 0.226 | 0.4455 | 13,472 |
| Our engine (crowd + deterministic correction) | 0.0704 [0.0678, 0.0734] | 0.2267 | 0.446 | 13,472 |
Calibration curve, as a table
A calibrated forecaster’s 30% calls should come true about 30% of the time. Each row groups the crowd’s matured forecasts by stated probability and shows what reality delivered — this is the resolved-outcome table, bin by bin.
| stated probability | resolved markets | mean prediction | came true | read |
|---|---|---|---|---|
| 0–10% | 8,252 | 2% | 1.4% | calibrated |
| 10–20% | 1,254 | 14.5% | 12.4% | calibrated |
| 20–30% | 1,047 | 24.6% | 20.3% | overconfident by 4.3% |
| 30–40% | 890 | 34.8% | 31.7% | overconfident by 3.1% |
| 40–50% | 700 | 44.7% | 32.1% | overconfident by 12.6% |
| 50–60% | 380 | 53.4% | 48.4% | overconfident by 5% |
| 60–70% | 143 | 64.2% | 57.3% | overconfident by 6.8% |
| 70–80% | 137 | 74.7% | 74.5% | calibrated |
| 80–90% | 112 | 85% | 81.3% | overconfident by 3.7% |
| 90–100% | 557 | 97% | 99.3% | calibrated |
The out-of-sample check
A second benchmark guards against fitting the past: 34,325 resolved markets split by resolution date, the engine fitted only on the earlier half and scored only on the later half, against a base-rate null.
| forecaster | Brier (95% CI) | log loss | skill vs base rate | markets |
|---|---|---|---|---|
| Base rate (the null) | 0.1063 [0.103, 0.1097] | 0.3699 | 0 | 17,163 |
| Crowd consensus | 0.0705 [0.0658, 0.0756] | 0.2339 | 0.5555 | 4,069 |
| Our engine | 0.07 [0.0652, 0.0751] | 0.2318 | 0.5589 | 4,069 |
The honest reading: the crowd is a strong, well-calibrated sensor — which is exactly why we cite it — and our engine’s deterministic correction adds a small, measured improvement on top. We publish the grade either way; the referee runs whether or not we like the score.
Reproducibility
The full dataset behind this page — every score, confidence interval, and calibration bin — ships as machine-readable JSON, regenerated daily as new markets mature.
Informational only — not financial, legal, or investment advice. Prediction-market prices are shown as a signal of what the crowd believes.