How much compute does it take to train an LLM? A widely used approximation is C ≈ 6 · N · D FLOPs, for N parameters trained on D tokens. The Chinchilla result says compute-optimal training uses roughly 20 tokens per parameter.
Implement:
training_flops(n_params, n_tokens): 6 * N * D.gpu_days(flops, gpu_flops_per_sec, utilization=0.4): wall-clock GPU-days at that fraction of peak (MFU).chinchilla_optimal(compute_flops): with D = 20N, solve C = 6 · N · 20N for N. Return (n_params, n_tokens) as floats.is_undertrained(n_params, n_tokens): True if there are fewer than 20 tokens per parameter.