Cerebras-GPT
Cerebras' fully open family of models trained on its wafer-scale chips
Cerebras-GPT is a family of language models, ranging from 111 million to 13 billion parameters, that Cerebras trained on its own CS-2 wafer-scale systems and released in March 2023. The point of the project was less about beating benchmarks and more about openness: Cerebras published the weights, training code, and a detailed technical report describing hyperparameters and scaling behavior for every size in the family, following compute-optimal training recipes similar to those behind Chinchilla. That made Cerebras-GPT one of the more fully documented open model families of its time, useful to researchers who wanted a clean, reproducible baseline for studying scaling laws.
All Cerebras-GPT checkpoints are released under Apache 2.0 and hosted on Hugging Face, free for commercial and research use. The models use a fairly standard GPT-style architecture with a 2,048-token context window, and by 2023 standards their raw capability was modest next to models like LLaMA, since Cerebras prioritized transparent, reproducible training over squeezing out maximum benchmark performance. The family is mainly cited today in scaling-law research and as an example of an AI hardware company demonstrating its own chips by training and open-sourcing a full model lineup.