Megatron-Turing NLG
NVIDIA and Microsoft's 530B-parameter research model, among the largest dense LLMs of its time
Megatron-Turing NLG is a 530 billion parameter dense transformer language model that NVIDIA and Microsoft trained jointly and announced in October 2021. At the time it was the largest monolithic (non-mixture-of-experts) language model ever trained, built by combining NVIDIA’s Megatron-LM tensor-parallelism with Microsoft’s DeepSpeed pipeline-parallelism and ZeRO optimizer across hundreds of A100 GPUs. The model has 105 transformer layers and was trained on a 270 billion token corpus drawn from web text, books, and other curated sources.
It served mainly as a research and engineering demonstration of how far dense-model scaling and distributed training software could go before mixture-of-experts architectures became the more practical path to larger models. NVIDIA and Microsoft published results on completion, reading comprehension, and reasoning benchmarks where it outperformed smaller predecessors like GPT-3 and Turing-NLG, but the model itself was never opened up as a public product or API.