A/B Test Design

Choosing what gets randomized — user, session, request, device, cluster, region — is an assertion about how effects propagate:

• User-level — clean, stable, the default; a user sees one consistent experience
• Session/request-level — more 'sample size' per user, but the same human sees both variants; carryover and inconsistency leak
• Cluster-level — randomize the interacting group (team, household, city) when units contaminate each other; power collapses to the cluster count
• Mismatch is the bug — randomizing users when effects flow through shared resources (marketplaces) doesn't reduce bias, it hides it

Rule: the unit must be independent under your interference model and aligned with the decision — you ship to users, so user-level metrics usually lead.