Vendor Evaluation & the Pilot Trap

A vendor demo is a curated performance: cherry-picked inputs, a warmed-up example set, edge cases surgically avoided, and nobody showing the latency histogram or the per-call cost. It answers 'can this ever work?' — which you already knew.

The only evaluation that matters runs on your data, at your volume, against your success metric. Ask for a trial on a few hundred of your real cases — ugly ones included — before believing any number. If the vendor resists your data, that resistance is the evaluation result.