Ways Experiments Lie

SRM is the most underused diagnostic in experimentation: if you designed a 50/50 split and observe 50.4/49.6 on a large sample, something upstream is broken — logging differences, bucketing bugs, bot filtering that hits one arm, redirect failures. The χ² check is trivial; the signal is invaluable.

Why it's first: an SRM means the randomization didn't deliver what it promised — every downstream comparison is suspect before you read a single metric. Check assignment integrity before treatment effects. Every time.