Roboflow Serverless Inference: A Thousand Models on a Shared GPU Fleet

The Challenge: Heavy Payloads, Unpredictable Compute, and Thousands of Models

To an external client, our serverless inference API looks like a standard web service: you POST an image and get back a clean JSON block of predictions, usually in under 100 milliseconds. Behind that simple promise sits some genuinely gnarly engineering. If your background is mostly in typical CRUD services, machine vision inference will break a few of your production mental models. Three traits do the breaking:

Ingress inversion. Most web services ingest kilobytes and emit the heavy bytes. Vision pipelines flip that: we ingest multi-megabyte, high-resolution frames and compute them down to a few kilobytes of detections. The expensive direction is inbound, and the whole pipeline has to be shaped to absorb that weight.

A 6,000× latency chasm. Latency here isn't a bell curve; it's aggressively bimodal. A model already warm in GPU memory answers in about 5ms. A cold one has to be fetched from the model registry, deserialized, and streamed into VRAM — routinely 30 seconds, and a minute or more for the largest foundation models. In production that means a median well under 100ms with 1–2% of requests landing past 10 seconds. There is no "typical" latency to design around: there's a fast lane and a slow lane, and you don't know which one a request is in until you're already committed.

VRAM multi-tenancy. We don't serve a static set of models; we host a spiky, long-tailed catalog of thousands of customer-trained models alongside heavyweight foundation ones. Only a small fraction of that catalog fits in GPU memory at once, so which models happen to be warm — and where — decides who gets the fast lane. Hold that thought; it shapes this design more than anything else on the list.

Why synchronous request-response fails

Serve this workload with a plain synchronous request-response API and four things go wrong, reliably — and the last one is the killer:

Timeouts fire at the worst moments. Clien…