Back-of-the-envelope capacity planning for an LLM endpoint. For a request with prompt_tokens and output_tokens, given a GPU's prefill speed (tokens/s processed for the prompt) and per-sequence decode speed (tokens/s generated):
prompt_tokens / prefill_tpsoutput_tokens / decode_tpsImplement request_latency(prompt_tokens, output_tokens, prefill_tps, decode_tps) returning {"ttft", "latency"} in seconds. Then implement gpus_needed(requests_per_sec, prompt_tokens, output_tokens, prefill_tps, decode_tps, concurrent_per_gpu): by Little's law, the average number of in-flight requests is requests_per_sec × latency. Divide by how many requests one GPU serves concurrently and round up.