The Hardware Landscape

Inside a modern phone SoC, inference can run on at least four different processors, each with a different contract:

• CPU — most flexible, lowest throughput; runs anything, slowly. The fallback for ops nothing else supports.
• GPU — high throughput on parallel float work; power-hungry but programmable. The workhorse for models without NPU support.
• DSP — Qualcomm's Hexagon and friends: efficient vector processors built for signal work; good integer performance at low power.
• NPU — dedicated matrix engines (Apple Neural Engine, Qualcomm's NPU, MediaTek's APU): highest efficiency on the ops they're designed for, useless on the ones they aren't.

Plus the edge outliers: Google's Coral TPU (~4 TOPS at ~2W — the embedded benchmark), and microNPUs like Arm's Ethos-U for microcontrollers. The art is matching ops to engines.