On-Device Runtimes

The model is weights; the runtime is everything else: the graph compiler that maps ops to hardware, the memory planner that fits tensors in budget, the scheduler that picks engines. Major runtimes by platform:

• TFLite / LiteRT — Android's standard: converts TF models, ships delegates (GPU, NPU via vendor drivers); strong for vision/audio
• CoreML — Apple's only path to the Neural Engine: compile to .mlpackage, supports palettization (weight-only quant) and op-level precision control
• ONNX Runtime Mobile — cross-platform with pluggable 'execution providers' per backend
• llama.cpp — the GGUF runtime: CPU-first with Metal/Vulkan backends; the de facto standard for on-device LLMs
• ExecuTorch / MediaPipe — PyTorch's edge runtime and Google's task-level wrappers

The runtime choice is architecture: it defines which ops accelerate, which formats you convert through, and which devices you actually reach.