Skip to content

Spec'ing the Probabilistic

Evals Are the New Specs

8 min+75 XPStep 1/11

Evals are the AI product requirement doc

In traditional products, the spec describes behavior and tests verify it. For AI features, the eval is the spec: a versioned set of inputs plus expected outputs (or grading rubrics) that defines 'good enough' before anyone builds.

'Summarize tickets accurately' is a wish. 'Given these 200 real tickets, summaries must capture all action items, invent zero facts, and score ≥4/5 on the rubric' is a spec you can iterate against.