LLM Evaluation & Observability

From vibes to metrics.

Golden datasets, LLM judges and their biases, rubric design, regression suites, and production tracing — the measurement stack that keeps a shipped model honest.

Advanced4 modules · 11 lessons~109 min1075 XPAligned with Braintrust/Anthropic eval practice + academic benchmarksStart with: Transformers & LLMs

Progress0/11

Module

Eval Foundations

Why LLMs broke the old testing model, the three graders you have, and the dataset that defines 'good'

Why LLM Evals Are Different9 min · 80 XPWhy LLM Evals Are Different

The Eval Toolbox: Code, Model, Human10 min · 90 XPThe Eval Toolbox: Code, Model, Human

Golden Datasets10 min · 90 XPGolden Datasets

Module

LLM-as-Judge

The judge has biases, the rubric is a spec, and the suite belongs in CI

The Judge Has Biases10 min · 100 XPThe Judge Has Biases

Rubrics That Grade Like You10 min · 100 XPRubrics That Grade Like You

Regression Suites & CI Evals9 min · 95 XPRegression Suites & CI Evals

Module

Production Observability

Traces over transcripts, the quality-cost-latency triangle, and the A/B test as the last honest eval

Tracing in Production10 min · 100 XPTracing in Production

Cost, Latency & the Quality Triangle10 min · 100 XPCost, Latency & the Quality Triangle

A/B Testing for Models10 min · 105 XPA/B Testing for Models

Module

When Evals Lie

Error taxonomies, Goodhart's law, contamination, and the discipline of distrusting your own measurements

Error Taxonomies: Name Your Failures10 min · 105 XPError Taxonomies: Name Your Failures

When Evals Lie11 min · 110 XPWhen Evals Lie

Capstone project

Grade the machine

Design an eval for an LLM task: pick a task, write 20 test cases with expected behavior, and score a real model against them. Where does it fail systematically?

Deliverable: Your test set, scoring results, and a one-paragraph failure analysis.

Milestones

Finish every lesson to earn the LLM Evaluation & Observability Certificate.