Batch, Real-Time, and the Space Between

Before touching infrastructure, two questions decide the topology: how stale can predictions be, and what latency does the caller need?

Batch scoring runs the model over the whole population on a schedule and writes results to a table — churn scores Sunday night, risk tiers every morning. Cheap, simple, throughput-optimized, and correct for most business decisions. Real-time serving is an endpoint that scores one request in milliseconds, because the user is staring at the screen right now — ad ranking, an ETA while the map is open.

Getting this wrong in either direction is expensive. Batch for a real-time problem means stale predictions; real-time infrastructure for a batch problem means paying a latency tax nobody asked for.