Cash Pop Late Night: Calibration Forge — measure false alarms and detection power

Loading…

Reading the draws and working out what chance predicts.

Before trusting a detector, make it earn its alarm.

Each method receives fresh fair histories to measure false alarms, then fresh rigged histories matched to the pattern it claims to detect. Calibration and evaluation simulations never overlap.

100 synthetic draws

Cash Pop Late Night1/15 era · since Apr 2022100 draws

Frequency χ²

static weighted balls

False alarms 5.0%95% interval 3.3%–7.6%Detection power 19.0%95% interval 15.5%–23.1%

Rank-stream KS

static weighted balls

False alarms 5.5%95% interval 3.7%–8.2%Detection power 46.0%95% interval 41.2%–50.9%

Locked hot strategy

learnable static bias

False alarms 10.0%95% interval 7.4%–13.3%Detection power 18.8%95% interval 15.2%–22.9%

Global e-process

persistent marginal bias

False alarms 5.8%95% interval 3.9%–8.5%Detection power 12.5%95% interval 9.6%–16.1%

Set-kernel MMD

abrupt regime change

False alarms 4.0%95% interval 2.5%–6.4%Detection power 5.8%95% interval 3.9%–8.5%

LZ rank stream

repeated draw motif

False alarms 7.8%95% interval 5.5%–10.8%Detection power 59.3%95% interval 54.4%–64.0%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

400 synthetic draws

Cash Pop Late Night1/15 era · since Apr 2022400 draws

Frequency χ²

static weighted balls

False alarms 4.0%95% interval 2.5%–6.4%Detection power 71.0%95% interval 66.4%–75.2%

Rank-stream KS

static weighted balls

False alarms 5.0%95% interval 3.3%–7.6%Detection power 94.5%95% interval 91.8%–96.3%

Locked hot strategy

learnable static bias

False alarms 8.3%95% interval 5.9%–11.4%Detection power 25.8%95% interval 21.7%–30.3%

Global e-process

persistent marginal bias

False alarms 4.8%95% interval 3.1%–7.3%Detection power 44.3%95% interval 39.5%–49.1%

Set-kernel MMD

abrupt regime change

False alarms 1.5%95% interval 0.7%–3.2%Detection power 6.3%95% interval 4.3%–9.1%

LZ rank stream

repeated draw motif

False alarms 10.5%95% interval 7.9%–13.9%Detection power 100.0%95% interval 99.0%–100.0%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

Every rate here is a proportion out of 400 seeded runs, so it carries a Wilson 95% interval and the interval is what should be read. At fifty runs — the figure this page used to quote — a nominal five per cent false-alarm rate had an interval running from about one per cent to fourteen, and a badly miscalibrated test looked exactly like a perfect one.

The maths, in plain language

Calibration before interpretation

For each detector and sample size, one bank of fair simulations estimates the 95th-percentile alarm threshold. Separate fair simulations estimate the false-positive rate, while separate rigged simulations estimate power.

$\widehat{\alpha}=\frac{\text{fair alarms}}{R},\qquad \widehat{\pi}=\frac{\text{rigged alarms}}{R}.$

A useful detector should keep $\widehat{\alpha}$ near 5% and have high $\widehat{\pi}$ for the departure it claims to see. Power is specific: a frequency test need not detect a repeated temporal motif, and a drift test need not detect a constant bias.

The story behind it

The fire alarm gets tested too

A detector that rings often will eventually look prophetic. Calibration asks how often it rings in an empty kitchen; power asks whether it rings when its particular fire is present.

This forge is deliberately synthetic and seeded. It makes no claim about the next draw. Its job is to expose weak instruments before they are pointed at history and given a story.

Before trusting a detector, make it earn its alarm.

Each method receives fresh fair histories to measure false alarms, then fresh rigged histories matched to the pattern it claims to detect. Calibration and evaluation simulations never overlap.

100 synthetic draws

Cash Pop Late Night1/15 era · since Apr 2022100 draws

Frequency χ²

static weighted balls

False alarms 5.0%95% interval 3.3%–7.6%Detection power 19.0%95% interval 15.5%–23.1%

Rank-stream KS

static weighted balls

False alarms 5.5%95% interval 3.7%–8.2%Detection power 46.0%95% interval 41.2%–50.9%

Locked hot strategy

learnable static bias

False alarms 10.0%95% interval 7.4%–13.3%Detection power 18.8%95% interval 15.2%–22.9%

Global e-process

persistent marginal bias

False alarms 5.8%95% interval 3.9%–8.5%Detection power 12.5%95% interval 9.6%–16.1%

Set-kernel MMD

abrupt regime change

False alarms 4.0%95% interval 2.5%–6.4%Detection power 5.8%95% interval 3.9%–8.5%

LZ rank stream

repeated draw motif

False alarms 7.8%95% interval 5.5%–10.8%Detection power 59.3%95% interval 54.4%–64.0%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

400 synthetic draws

Cash Pop Late Night1/15 era · since Apr 2022400 draws

Frequency χ²

static weighted balls

False alarms 4.0%95% interval 2.5%–6.4%Detection power 71.0%95% interval 66.4%–75.2%

Rank-stream KS

static weighted balls

False alarms 5.0%95% interval 3.3%–7.6%Detection power 94.5%95% interval 91.8%–96.3%

Locked hot strategy

learnable static bias

False alarms 8.3%95% interval 5.9%–11.4%Detection power 25.8%95% interval 21.7%–30.3%

Global e-process

persistent marginal bias

False alarms 4.8%95% interval 3.1%–7.3%Detection power 44.3%95% interval 39.5%–49.1%

Set-kernel MMD

abrupt regime change

False alarms 1.5%95% interval 0.7%–3.2%Detection power 6.3%95% interval 4.3%–9.1%

LZ rank stream

repeated draw motif

False alarms 10.5%95% interval 7.9%–13.9%Detection power 100.0%95% interval 99.0%–100.0%

Independent null calibration and power simulationcalibration-bench-v2seed 20260827400 calibration + 400 evaluation histories per cell

Gold is useful detection power. The outlined meter is false-positive rate; its target is 5%.

The maths, in plain language

Calibration before interpretation

$\widehat{\alpha}=\frac{\text{fair alarms}}{R},\qquad \widehat{\pi}=\frac{\text{rigged alarms}}{R}.$

The story behind it

The fire alarm gets tested too

A detector that rings often will eventually look prophetic. Calibration asks how often it rings in an empty kitchen; power asks whether it rings when its particular fire is present.

This forge is deliberately synthetic and seeded. It makes no claim about the next draw. Its job is to expose weak instruments before they are pointed at history and given a story.