Skip to content
Data Science & Analytics

Data Cleaning

Duplicates & Leakage

8 min+75 XPStep 1/10

The same row, wearing disguises

Deduplication sounds trivial until you meet real data. A retail export holds 'Acme Inc.', 'ACME INC', and 'Acme, Inc.' — same customer, three spellings. DISTINCT catches none of them.

Duplicates come in layers:

  • Exact: the export ran twice — identical rows. Easy: a unique key or DISTINCT.
  • Fuzzy: same entity, different strings — needs normalization (lowercase, strip punctuation) or matching on email and phone instead of name.
  • Semantic: two rows, one event — a transaction logged once per system. Hardest to spot; check timestamps and IDs.

Every surviving duplicate gets double-counted in each sum, average, and churn rate you compute.

warning

Two customers with the same name are not a duplicate; one customer with two spellings is. Deduplicate on identity — an ID, an email — not on the label.