ChatGPT: one token at a time, for a billion people

How an LLM serving stack streams replies to over a billion weekly users: count tokens and GPU memory instead of requests, batch hundreds of replies into every decode step with continuous batching and paged KV memory, route conversations to their cached prefix, never pay the GPU twice for a dropped stream, and choose fairly when demand beats capacity.