Evidence, Not Demos
Why Accuracy Isn't the Metric
10 min+100 XPStep 1/11
Accuracy is the wrong yardstick for risk
A model that's '97% accurate' on a 1%-prevalence condition may be calling everything negative — accuracy hides the confusion matrix. Even sensitivity/specificity and AUROC miss the point: they average over all thresholds, but a clinic operates at exactly one.
The questions that matter are clinical: at the operating threshold, what's the positive predictive value (how many flagged patients are real), the alert burden (how many false alarms per true catch), and — deepest — net benefit: does acting on this output beat treat-all, treat-none, and current practice?
insight
A model can have worse AUROC and better clinical value if its calibration puts probability mass where decisions actually happen.