Skip to content
Causal Inference & Experimentation

Experimentation Engineering

Power, Sample Size & the MDE

10 min+100 XPStep 1/10

Four knobs, one equation

Power analysis trades off four quantities — fix three, the fourth is determined:

  • α — false-positive rate (typically 5%): how often you declare phantom effects
  • Power (1−β, typically 80%): how often you catch a real effect of size MDE
  • MDE — minimum detectable effect: the smallest true lift your test can reliably catch
  • n — sample size needed

The uncomfortable arithmetic: power halves faster than intuition expects as MDE shrinks; detecting a 1% relative lift needs ~an order of magnitude more traffic than 10%. Underpowered tests don't 'probably still work' — they mostly miss, and their significant results are disproportionately inflated (the winner's curse).

insight

A test underpowered for the effect size that matters is a noise generator with a p-value attached. The significant results it produces are disproportionately false or inflated.