gpt-oss

OpenAI · 2025-08-05 · analysed 2026-08-10 · skill v0.3.5

Technical report ↗

Design overview

gpt-oss-120b and gpt-oss-20b are OpenAI's first open-weight MoE release: 36-layer (24-layer) decoder-only transformers with a 2880-wide residual stream, RMSNorm with Pre-LN placement, and an explicit GPT-2/GPT-3 lineage. Every MLP block is a router MoE — 128 experts (120b) or 32 (20b) with a linear router selecting the top-4 experts per token, outputs weighted by a softmax over the selected experts only, using a gated SwiGLU activation that (unusually) includes clamping and an internal residual connection. Attention follows the GPT-3 sparse-transformer pattern, alternating banded-window (128-token bandwidth) and fully dense layers; each layer has 64 query heads of dimension 64 with GQA over 8 KV heads, RoPE, and a learned per-head bias in the softmax denominator (off-by-one attention / attention-sinks style) that lets a head attend to nothing. Dense layers are extended to 131,072 tokens via YaRN. The card's distinctive engineering is deployment-focused: MoE weights are quantized post-training to MXFP4 (4.25 bits/parameter), fitting 120b on a single 80GB GPU and 20b in ~16GB, and the o200k_harmony tokenizer adds chat-format tokens. Post-training is o3-style CoT RL with three configurable reasoning levels, a harmony chat format with a System > Developer > User > Assistant > Tool hierarchy, and agentic tool use (browse, Python, developer functions). As a model card, it discloses far less architecture than technical reports of comparable MoE peers.

What the report claims is novel

• Introduces gpt-oss-120b and gpt-oss-20b, two open-weight reasoning models under the Apache 2.0 license with full chain-of-thought, Structured Outputs, and adjustable reasoning effort [§1]
• MoE with top-4 routing: 128 experts (120b) / 32 experts (20b), gated SwiGLU, and softmax weighting over only the selected experts [§2.2]
• GPT-3-style alternating banded-window (bandwidth 128) and fully dense attention with GQA (64 Q / 8 KV heads, head dim 64), RoPE, and a learned per-head softmax-denominator bias (off-by-one attention / attention sinks) enabling attention to pay no attention to any tokens; dense layers extended to 131,072 tokens via YaRN [§2.2]
• MXFP4 post-training quantization of MoE weights (4.25 bits per parameter) fitting gpt-oss-120b on a single 80GB GPU and gpt-oss-20b in ~16GB memory [§2.1]
• o200k_harmony tokenizer open-sourced in the TikToken library, extending the o200k tokenizer with harmony chat format tokens (201,088 total) [§2.3]
• Harmony chat format with role-based instruction hierarchy (System > Developer > User > Assistant > Tool) and channels marking CoT, commentary, and final-answer visibility for agentic workflows [§2.5.1]
• Safety: Preparedness Framework evaluations including adversarially fine-tuned versions of gpt-oss-120b (SAG-reviewed), concluding the model does not reach High capability in Biological/Chemical or Cyber risk [§1, §5]

Atlas assessment: A conventional decoder-only MoE in the GPT-2/GPT-3 lineage with two distinctive touches: the revived GPT-3 sparse-transformer pattern (alternating banded-window/dense attention, here with GQA, RoPE and YaRN extension) and a learned per-head softmax-denominator bias for 'attend to nothing' (off-by-one attention family). vs DeepSeek-V3, the closest atlas MoE peer: far shallower (36 vs 61 layers), far fewer active params (5.1B vs 37B), top-4 of 128 experts with no shared expert, no MLA, no disclosed load balancing, and no MTP; the report also lacks any MoE training-stability discussion. The MXFP4-native release — quantization applied during post-training at 4.25 bits/parameter and treated as a first-class artifact (checkpoint sizes given in GiB) — is distinctive among atlas entries. The reasoning post-training (o3-style CoT RL with variable low/medium/high effort) extends the deepseek-r1 RLVR line with an explicit test-time effort control that no other atlas record implements; the model card's safety apparatus (adversarial fine-tuning simulations) is unique in the atlas but not architecture.

Relation to the atlas

Lineage in

• Autoregressive MoE transformer architecture building upon the GPT-2 and GPT-3 architectures from GPT-2 / GPT-3
• Alternating banded-window and fully dense attention layers from GPT-3 / Sparse Transformer
• Grouped Query Attention (GQA, 64 Q / 8 KV heads) from GQA (Ainslie et al. 2023)
• Rotary position embeddings (RoPE) from RoFormer (Su et al.)
• YaRN context extension to 131,072 tokens from YaRN (Peng et al. 2023)
• Learned per-head bias in the softmax denominator (off-by-one attention / attention sinks) from Off-by-one attention (Miller 2023) / attention sinks (Xiao et al. 2023)
• CoT reinforcement-learning post-training techniques from OpenAI o3
• o200k tokenizer base extended with harmony chat format tokens (o200k_harmony) from o200k (GPT-4o / o4-mini)

Influence out

• xiaomi-mimo-v2-flash-309b: Learnable attention sink bias in the softmax denominator

Notable omissions

• No pre-training token count (only 'trillions of tokens', §2.4)
• No pre-training sequence length; only the YaRN-extended 131,072-token dense-layer context is given
• Per-expert MoE inner dimension not disclosed (ffn_inner_dim / ffn_ratio unknown)
• No MoE load-balancing mechanism disclosed
• No RoPE base frequency, partial-RoPE, or YaRN factor disclosed
• Master training precision, optimizer, LR and batch schedules, and parallelism not disclosed
• Input/output embedding tying not stated (only a joint 'Embed + Unembed' 1.16B count, Table 1)
• RL algorithm beyond 'CoT RL similar to o3' never named
• No data mixture percentages and no full license text for the gpt-oss usage policy
• No architectural ablations or scaling-law studies (safety-focused model card)
• Benchmark comparisons limited to OpenAI's own o3 / o3-mini / o4-mini family; no independent third-party leaderboards

Closest relatives in the atlas

• Trinity — 8 shared comparable fields
• Mellum 2 — 7 shared comparable fields
• DeepSeek-V4 — 7 shared comparable fields

Spec sheet

Identity

Variants

• 20B
• 120B

Scale

Per-variant configs

Core block

Context

Tokenizer

Training

Post-training

Modality

Inference efficiency

Evaluation

Benchmarks (report's own numbers only)

Provenance references

1. stated · Title ('gpt-oss-120b & gpt-oss-20b Model Card')
2. stated · Title page (OpenAI)
3. stated · Title page ('August 5, 2025')
4. stated · pipeline
5. stated · §1 ('available under the Apache 2.0 license and our gpt-oss usage policy')
6. stated · §1 ('two open-weight reasoning models available under the Apache 2.0 license')
7. stated · Table 1
8. stated · §2
9. stated · §2.2 ('Both models have a residual stream dimension of 2880')
10. stated · §2.2 ('Each layer has 64 query heads of dimension 64')
11. stated · §2.2 ('Grouped Query Attention (GQA) with 8 key-value heads')
12. stated · §2.2 ('64 query heads of dimension 64')
13. stated · §2.3 ('a total of 201,088 tokens')
14. stated · §2 ('The gpt-oss models are autoregressive Mixture-of-Experts (MoE) transformers')
15. stated · §2.2
16. stated · §2.2 ('a fixed number of experts (128 for gpt-oss-120b and 32 for gpt-oss-20b)')
17. stated · §2.2 ('we select the top-4 experts for each token')
18. stated · §2.2 ('We apply rotary position embeddings')
19. stated · §2.2 ('applying root mean square normalization [6] on the activations before each attention and MoE block')
20. stated · §2.2 ('Similar to GPT-2 we use Pre-LN placement')
21. stated · §2.3
22. stated · §2.4
23. stated · §2.5
24. stated · §2.5, §3
25. stated · §1 ('these text-only models')
26. n/a · §1
27. stated · §2.1, Table 1