Eval Foundations
Golden Datasets
10 min+90 XPStep 1/10
The dataset is the product spec, made concrete
A golden dataset is a curated set of inputs with expected outputs or grading criteria — the eval suite's foundation. What makes it golden isn't size, it's coverage: does it contain the cases that decide whether the system works?
The best source isn't imagination — it's production. Logged queries, sampled traces, labeled failures: real distribution, real edge cases, real adversarial inputs. A golden set built from 'what we thought users would ask' is a model of your assumptions; one built from traffic is a model of the world.
insight
The eval set that matters is built from what actually happened — production traces, sampled and labeled, are a better spec than anything a team invents in a meeting.