Calibration Forge

Before trusting a detector, make it earn its alarm.

Each method receives fresh fair histories to measure false alarms, then fresh rigged histories matched to the pattern it claims to detect. Calibration and evaluation simulations never overlap.

100 synthetic draws

Georgia FIVE5/9 era · since Apr 2012100 draws

Frequency χ²

static weighted balls

False alarms 6.0%95% interval 4.1%–8.8%Detection power 90.5%95% interval 87.2%–93.0%

Rank-stream KS

static weighted balls

False alarms 5.0%95% interval 3.3%–7.6%Detection power 66.3%95% interval 61.5%–70.7%

Locked hot strategy

learnable static bias

False alarms 5.8%95% interval 3.9%–8.5%Detection power 81.0%95% interval 76.9%–84.5%

Global e-process

persistent marginal bias

False alarms 5.5%95% interval 3.7%–8.2%Detection power 73.3%95% interval 68.7%–77.4%

Set-kernel MMD

abrupt regime change

False alarms 4.5%95% interval 2.9%–7.0%Detection power 19.0%95% interval 15.5%–23.1%

LZ rank stream

repeated draw motif

False alarms 13.5%95% interval 10.5%–17.2%Detection power 83.0%95% interval 79.0%–86.4%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

400 synthetic draws

Georgia FIVE5/9 era · since Apr 2012400 draws

Frequency χ²

static weighted balls

False alarms 5.8%95% interval 3.9%–8.5%Detection power 100.0%95% interval 99.0%–100.0%

Rank-stream KS

static weighted balls

False alarms 4.0%95% interval 2.5%–6.4%Detection power 100.0%95% interval 99.0%–100.0%

Locked hot strategy

learnable static bias

False alarms 4.3%95% interval 2.7%–6.7%Detection power 100.0%95% interval 99.0%–100.0%

Global e-process

persistent marginal bias

False alarms 6.0%95% interval 4.1%–8.8%Detection power 100.0%95% interval 99.0%–100.0%

Set-kernel MMD

abrupt regime change

False alarms 5.5%95% interval 3.7%–8.2%Detection power 16.3%95% interval 13.0%–20.2%

LZ rank stream

repeated draw motif

False alarms 7.8%95% interval 5.5%–10.8%Detection power 100.0%95% interval 99.0%–100.0%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

Every rate here is a proportion out of 400 seeded runs, so it carries a Wilson 95% interval and the interval is what should be read. At fifty runs — the figure this page used to quote — a nominal five per cent false-alarm rate had an interval running from about one per cent to fourteen, and a badly miscalibrated test looked exactly like a perfect one.

The maths, in plain language

Calibration before interpretation

For each detector and sample size, one bank of fair simulations estimates the 95th-percentile alarm threshold. Separate fair simulations estimate the false-positive rate, while separate rigged simulations estimate power.

$\widehat{\alpha}=\frac{\text{fair alarms}}{R},\qquad \widehat{\pi}=\frac{\text{rigged alarms}}{R}.$

A useful detector should keep $\widehat{\alpha}$ near 5% and have high $\widehat{\pi}$ for the departure it claims to see. Power is specific: a frequency test need not detect a repeated temporal motif, and a drift test need not detect a constant bias.

The story behind it

The fire alarm gets tested too

A detector that rings often will eventually look prophetic. Calibration asks how often it rings in an empty kitchen; power asks whether it rings when its particular fire is present.

This forge is deliberately synthetic and seeded. It makes no claim about the next draw. Its job is to expose weak instruments before they are pointed at history and given a story.

Loading…

Reading the draws and working out what chance predicts.

Before trusting a detector, make it earn its alarm.

Each method receives fresh fair histories to measure false alarms, then fresh rigged histories matched to the pattern it claims to detect. Calibration and evaluation simulations never overlap.

100 synthetic draws

Georgia FIVE5/9 era · since Apr 2012100 draws

Frequency χ²

static weighted balls

False alarms 6.0%95% interval 4.1%–8.8%Detection power 90.5%95% interval 87.2%–93.0%

Rank-stream KS

static weighted balls

False alarms 5.0%95% interval 3.3%–7.6%Detection power 66.3%95% interval 61.5%–70.7%

Locked hot strategy

learnable static bias

False alarms 5.8%95% interval 3.9%–8.5%Detection power 81.0%95% interval 76.9%–84.5%

Global e-process

persistent marginal bias

False alarms 5.5%95% interval 3.7%–8.2%Detection power 73.3%95% interval 68.7%–77.4%

Set-kernel MMD

abrupt regime change

False alarms 4.5%95% interval 2.9%–7.0%Detection power 19.0%95% interval 15.5%–23.1%

LZ rank stream

repeated draw motif

False alarms 13.5%95% interval 10.5%–17.2%Detection power 83.0%95% interval 79.0%–86.4%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

400 synthetic draws

Georgia FIVE5/9 era · since Apr 2012400 draws

Frequency χ²

static weighted balls

False alarms 5.8%95% interval 3.9%–8.5%Detection power 100.0%95% interval 99.0%–100.0%

Rank-stream KS

static weighted balls

False alarms 4.0%95% interval 2.5%–6.4%Detection power 100.0%95% interval 99.0%–100.0%

Locked hot strategy

learnable static bias

False alarms 4.3%95% interval 2.7%–6.7%Detection power 100.0%95% interval 99.0%–100.0%

Global e-process

persistent marginal bias

False alarms 6.0%95% interval 4.1%–8.8%Detection power 100.0%95% interval 99.0%–100.0%

Set-kernel MMD

abrupt regime change

False alarms 5.5%95% interval 3.7%–8.2%Detection power 16.3%95% interval 13.0%–20.2%

LZ rank stream

repeated draw motif

False alarms 7.8%95% interval 5.5%–10.8%Detection power 100.0%95% interval 99.0%–100.0%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

The maths, in plain language

Calibration before interpretation

$\widehat{\alpha}=\frac{\text{fair alarms}}{R},\qquad \widehat{\pi}=\frac{\text{rigged alarms}}{R}.$

The story behind it

The fire alarm gets tested too

A detector that rings often will eventually look prophetic. Calibration asks how often it rings in an empty kitchen; power asks whether it rings when its particular fire is present.

This forge is deliberately synthetic and seeded. It makes no claim about the next draw. Its job is to expose weak instruments before they are pointed at history and given a story.