ALBERT
Google's parameter-sharing variant of BERT designed for efficiency at scale
ALBERT (A Lite BERT) is a language representation model that Google Research and Toyota Technological Institute published in September 2019 as a follow-up to BERT. It uses two parameter-reduction tricks, factorized embedding parameterization and cross-layer parameter sharing, to cut the parameter count sharply while keeping accuracy competitive with much larger models. The largest configuration, ALBERT-xxlarge at around 223 million parameters, has roughly 18 times fewer parameters than BERT-large yet trains faster and set new state-of-the-art scores on GLUE, SQuAD 2.0, and RACE at the time of release.
Like BERT, ALBERT is an encoder-only model meant for understanding tasks such as classification, question answering, and reading comprehension rather than open-ended text generation, and it caps input at 512 tokens per sequence. Google released the code and pretrained weights openly, and the model became a common baseline for NLP research and production systems in the early 2020s before decoder-only generative models took over most attention in the field.