GPT-2 is a decoder-only transformer stack with learned absolute positional embeddings, pre-norm LayerNorm placement, a byte-level BPE vocabulary, and a 1024-token context. The report's contribution is largely empirical: scaling a plain transformer to 1.5B parameters and showing zero-shot transfer across 8 of the 9 evaluated task families without task-specific fine-tuning. Architecturally it is the direct ancestor of the atlas's dense decoder line — same block shape that Llama 3.1, Phi-3 and OLMo 2 later refine with GQA, RoPE, SwiGLU and modern training recipes. Its historical interest in the atlas is as the baseline against which everything newer is measured: no KV-cache optimisation, no grouped attention, no positional-encoding extension story, and no disclosed compute or optimizer details.
What the report claims is novel
Language models as unsupervised multitask learners: zero-shot task transfer without task-specific supervision [Abstract / §1]
WebText corpus construction (outbound Reddit links, ≥3 karma) [§2.1]
Byte-level BPE with no cross-category merges enabling universal encoding [§2.2]
Atlas assessment: Pioneering record relative to the atlas: predates every other entry; established the decoder-only pretrain-then-zero-shot paradigm that the atlas's dense line (Llama 3.1, Phi-3) descends from; within-atlas novelty comparisons do not apply
Relation to the atlas
Lineage in
decoder-only transformer with pre-norm and residual connections from GPT (Radford et al. 2018)
Influence out
openai-gpt-oss: Autoregressive MoE transformer architecture building upon the GPT-2 and GPT-3 architectures
Notable omissions
No attention head counts, FFN inner dimensions, optimizer, learning rate, precision, hardware or compute disclosed
Positional-encoding method never named (era-default learned absolute embeddings not stated)
No KV-cache / serving discussion (report predates the concern)
Weights release mentioned only as 'preliminary code' for the small model
Report undated and not on arXiv — release_date unknown