DeepSeek-V2
DeepSeek's 2024 mixture-of-experts model that undercut rivals on API price
DeepSeek-V2 is a mixture-of-experts language model that DeepSeek released in May 2024, with 236 billion total parameters and 21 billion active per token. Its main technical contribution was Multi-head Latent Attention, a method for compressing the key-value cache that cut memory use dramatically and let the model serve far more tokens per second than DeepSeek’s earlier dense 67B model on the same hardware. Trained on 8.1 trillion tokens with a 128K context window, it scored competitively on MMLU and coding benchmarks against other open models of the time.
What made V2 notable wasn’t just its architecture but its price: DeepSeek launched its API at a fraction of what Chinese and Western rivals were charging, kicking off a round of price cuts across the industry that commentators at the time called a “price war.” The MLA and DeepSeekMoE techniques introduced here carried forward into every subsequent DeepSeek model, including V3 and R1, making V2 the architectural foundation for the company’s later frontier releases.