GPT-2

GPT-2 is OpenAI’s 1.5-billion-parameter language model, first previewed in a February 2019 blog post and released in full on November 5, 2019. Unlike a normal release, OpenAI staged it over nine months: a 124-million-parameter checkpoint in February, a 355-million version in May, a 774-million version in August, and finally the full 1.5-billion model in November, alongside its complete code. The model uses a 1,024-token context window and a 50,257-token vocabulary, and was trained on WebText, a corpus scraped from outbound Reddit links.

OpenAI’s initial decision to withhold the largest checkpoint drew significant attention because it was one of the first times an AI lab cited misuse risk, specifically the potential for automated generation of convincing fake text, as a reason to delay a release. In hindsight the withholding period is remembered as much for the precedent it set around responsible disclosure as for GPT-2’s own capabilities, which look modest next to later GPT models but were striking at the time for their fluency and coherence.