Methodology

Our goal is to provide a transparent, repeatable way to evaluate probabilistic forecasts. We use the Brier Score as our primary metric because it is a strictly proper scoring rule. That means: given a fixed set of questions and fixed scoring rules, your best long run strategy is to report your true probabilities. However, leaderboards can still be gamed through selection (which questions you forecast) and timing (when you forecast). That is why we publish checkpoint rules, coverage, and benchmark definitions below.

---

0. What we score (the scorecard rules)

A score is only meaningful if the rules are clear. Our scorecards disclose:

• Eligible set: which markets or questions are included (and which are excluded).
• Checkpoint rule: which forecast is scored (to avoid rewarding last-minute updates).
• Weighting: how forecasts are weighted (default: equal weight per scored market).
• Benchmarks: what baseline we compare against for skill scoring.
• Reporting: sample size (N), coverage, and calibration diagnostics.

1. The Brier Score

The Brier Score measures how close your probabilities were to outcomes for binary events (YES/NO). It is the mean squared error of probability forecasts.

• p (Forecast): your probability for YES (e.g., 0.75).
• o (Outcome): 1 if YES happened, 0 if NO happened.
• Range: from 0.00 (perfect) to 1.00 (max error).

Lower is always better. Always forecasting 0.50 yields a Brier score of 0.25. (This is a useful baseline for balanced, 50/50 style questions.)

2. Timing and checkpoints (which forecast gets scored)

If you score only the last update right before settlement, you mostly reward late information. To make comparisons fair, we use an evaluation checkpoint rule on scorecards that need it.

• Multiple updates: only the forecast that exists at the checkpoint is scored.
• No look-ahead: benchmarks and consensus snapshots must be taken at or before the checkpoint time.

3. Coverage and minimum activity

Strong scores on tiny samples are usually noise. Every scorecard shows:

• N (sample size): number of scored forecasts.
• Coverage: how many eligible markets you participated in, not just the ones you picked.

Coverage matters because selection bias can make almost anyone look great if they only forecast easy questions.

4. Brier Skill Score (BSS)

Raw Brier Score is hard to interpret in isolation. A score of 0.15 can be amazing or average depending on how easy the questions were. Brier Skill Score compares you to a defined baseline.

• BSS > 0: better than the baseline.
• BSS = 0: same as the baseline.
• BSS < 0: worse than the baseline.

5. Benchmarks (what baseline means here)

We use benchmarks that are explicit and reproducible:

6. Calibration vs sharpness

A strong Brier score usually comes from two components:

7. Reference

Glenn W. Brier, "Verification of Forecasts Expressed in Terms of Probability", Monthly Weather Review, 1950.

If you want to replicate our results, the key is to use the same eligible set, checkpoint rule, and benchmark definitions. If any of these change, scores are no longer comparable.