Experimentation Engineering
CUPED & Variance Reduction
Variance you already paid for
Users differ enormously in baseline behavior — some whales, some ghosts. That heterogeneity is variance in your outcome metric, and it drowns small treatment effects. CUPED (Controlled-experiment Using Pre-Experiment Data, Microsoft 2013) removes it: adjust each user's outcome by how predictable it was from pre-experiment data.
The formula is a regression in one line: Y' = Y − θ·(X − E[X]), where X is pre-period behavior and θ is the correlation slope. If last month's activity predicts this month's — and it does — subtracting the predictable component shrinks residual variance. Reported variance cuts of 30-50% are common: equivalent to doubling-to-quadrupling your traffic, free.
insight
CUPED is the cheapest sample-size upgrade in experimentation: the 'traffic' it adds is pre-experiment data you were already storing.