Chinchilla

Chinchilla is a research language model that DeepMind described in a March 2022 paper on compute-optimal training. At 70 billion parameters, it is much smaller than DeepMind’s earlier 280 billion parameter Gopher model, yet Chinchilla was trained on roughly four times more data and beat Gopher, GPT-3, and other larger models of the era on a wide range of benchmarks, including an average MMLU score around 67.5%. The paper’s core finding was that most large language models up to that point were undertrained relative to their size: for a fixed compute budget, model size and training data should scale together, not just model size alone.

That result, now widely known as the Chinchilla scaling law, reshaped how labs planned training runs afterward, pushing the field toward smaller models trained on much larger datasets instead of simply growing parameter counts. DeepMind never released Chinchilla’s weights or offered any public access to it; it exists only as a research artifact described in the paper. Its influence shows up indirectly in nearly every major model trained since, from LLaMA to GPT-4, all of which cite compute-optimal scaling as part of their training methodology.