Skip to content
Reading AI Research

How to Read a Paper

Red Flags in ML Papers

9 min+90 XPStep 1/10

The baseline shell game

The most common trick in ML papers: compare against weak baselines. Beating a two-year-old competitor by 1% proves far less than matching the current best with a simpler method.

Questions to ask of any results table:

  • Are the baselines recent and tuned, or strawmen? Check their publication dates.
  • Is the evaluation setup matched? A method tested with extra test-time augmentation against baselines without it is running a rigged race.
  • Did they pick the benchmarks where they win? Evaluating on seven datasets and winning two isn't dominance — it's selection.

warning

The easiest way to beat state-of-the-art is to choose which state-of-the-art you're comparing against.