Nanbeige4.2-3B is a 3B non-embedding-parameter (4B total) dense model whose one structural idea is depth on a budget: a Looped Transformer that passes hidden states through the same Transformer stack twice, roughly doubling effective computational depth without adding parameters. The report is explicit that the loop is trained from scratch (upcycling a standard transformer into a loop tested worse), that two passes is the sweet spot (retaining about 75% of a standard transformer's token efficiency for a significant capacity gain), and that KV-cache sharing across passes was rejected: it halves inference KV memory but consistently underperforms the full loop, so the shipped model keeps separate caches per pass. Beyond that single architectural decision, the report discloses almost no block details — no layer count, hidden width, heads, vocabulary, normalization, activation, or positional encoding — and concentrates instead on the post-training recipe that makes a 3B model agentic: a three-stage SFT curriculum that grows context 64K → 128K → 256K while shifting the target-token mixture from 82.7% STEM reasoning to 68.9% agentic data; turn-level loss masking so the model learns recovery from failed turns; two-stage RLHF over Think and Non-Think responses with a point-wise reward model; reasoning RL with problem-dependent length budgets and a difficulty-aware penalty; and agentic RL whose action-centric rubrics feed turn-level process rewards alongside outcome rewards, cutting the action error rate by about 20%. The result is positioned as a compact general agent, deployable locally (OpenClaw), that beats Qwen3.5-9B and Gemma4-12B on the reported agentic suites.
What the report claims is novel
First open-source model at this scale (3B non-embedding) combining code-agent, office-agent, and complex tool-use capabilities while retaining strong general reasoning [§1]
Looped Transformer trained from scratch (two-pass loop reusing the layer stack), shown to outperform upcycling-based loop conversion; loop depth and KV-cache-sharing configurations studied [§2.1]
Hybrid environment pool (real + synthesized) with execution-grounded trajectory synthesis across heterogeneous scaffolds (Claude Code, OpenHands, SWE-agent, Codex-based drivers) to induce scaffold-invariant strategies [§3.1]
Two-stage RLHF over Think and Non-Think responses, with findings of cross-task and cross-mode generalization of behavioral regularization (repetition/format/termination control) [§3.2.2]
Reasoning RL with a problem-dependent length-control objective (offline median-length budgets + difficulty-aware penalty) balancing accuracy and reasoning efficiency [§3.2.3]
Agentic RL with action-centric rubrics as turn-level process rewards (tool-call accuracy, per-turn information gain), stabilizing long-horizon training and cutting the single-turn action error rate by ~20% [§3.2.4]
Atlas assessment: The block-level differentiator is the two-pass Looped Transformer: no other atlas record reuses its own layer stack for a second pass to gain effective depth without parameters (the report cites the Loopy line of work, not in the atlas). Everything else — SFT curriculum, two-stage RLHF on Think/Non-Think, length-controlled reasoning RL, rubric-based agentic RL — is recipe work similar in spirit to the mini-activation agentic post-training of minimax-m2 but at 3B scale and with the length-control twist. The report is honest about its scope: it discloses almost no architecture hyperparameters, so the loop's exact depth/width and attention configuration cannot be compared against atlas peers. The claimed 'first at this scale' positioning (code + office + tool-use agents with retained reasoning) is plausible but not independently verifiable from the report.
Relation to the atlas
Lineage in
Looped Transformer — reusing the shared layer stack for an additional pass (two-pass loop) from Looped Transformers (Bae et al.; not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
No architecture hyperparameters disclosed (layers, hidden width, heads, vocab, FFN dim, normalization, activation, positional encoding) despite 'we tune the depth of the transformer stack and its hidden width'
No optimizer, learning-rate schedule, batch size, precision, parallelism, hardware, or compute disclosure
Tokenizer not described
No license named
No release date printed (undated preprint)
Pre-training context length not separately disclosed (only the SFT curriculum reaching 256K)
The 4.1 → 4.2 delta is not quantified beyond loop architecture, data scale, and recipe changes
RL algorithm details (which policy-gradient method underlies the reasoning/agentic RL stages) not named
No embedding-tying or KV-cache-size statements for the underlying attention
three-stage SFT curriculum extending the maximum training context 64K → 128K → 256K (training at the deployed length); no positional-encoding extension method described[11]
corpus exceeding Nanbeige 4.1 in scale and quality; increased sampling weights for mathematics, code, and synthetic QA data (found particularly beneficial for compact models); a small proportion of agentic trajectory data mixed into pre-training ('an initial step toward agentic pre-training')[13]
Curriculum / staging
single-stage from-scratch pre-training on 28T tokens (no staging disclosed); post-pre-training, the SFT recipe runs a three-stage curriculum (64K → 128K → 256K context) that shifts the supervised target-token mixture from STEM-centered reasoning (82.7% at 64K) through a transition stage (47.8% STEM / 29.5% agentic at 128K) to agentic-dominated (68.9% agentic / 22.4% STEM / 8.7% general at 256K)[14]
Think/Non-Think hybrid modes with two-stage RLHF; reasoning RL with a problem-dependent length-control objective (offline median-length budgets per problem, difficulty-aware penalty alternating constrained and free-expansion phases); agentic RL with action-centric rubrics providing turn-level process rewards plus outcome rewards, restricted to easy tasks (short trajectories, high pass@8) at this scale[17]
full (non-sharing) two-pass loop: each pass through the shared layer stack keeps its own KV cache, so inference KV memory is roughly doubled versus a single-pass stack; the report explicitly rejected KV-cache sharing across loop passes because it halves the cache but reduces performance
Quantisation shipped
none disclosed
Speculative decoding
none disclosed
Serving optimisations
none disclosed
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
GDPval Rubrics
74.3
Table 3
AgentIF-Oneday
67.5
Table 3
OfficeQA-Pro
21.1
Table 3
PinchBench-V2
74.7
Table 3
ClawGym
65.0
Table 3
Claw-Eval Pass^3
52.2
Table 3
MCP-Atlas
57.8
Table 3
SWE-Bench Verified
63.6
Table 3
SWE-Bench Pro
46.9
Table 3
Terminal-Bench 2.0
44.1
Table 3
HLE (w/o Search)
17.8
Table 3
SciCode
35.6
Table 3
GPQA Diamond
87.4
Table 3
HMMT-Feb-2026
82.8
Table 3
IMO-Answer-Bench
67.3
Table 3
LiveCodeBench-V6
72.5
Table 3
AA-LCR
58.7
Table 3
IF-Bench
54.6
Table 3
Recruit-Bench
63.3
Table 3
Provenance references
stated · Title ('Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model')
stated · Title page byline
stated · pipeline
stated · Abstract ('The model checkpoint is available at https://huggingface.co/Nanbeige/Nanbeige4.2-3B'); §1 ('We open-source Nanbeige4.2-3B')
stated · Abstract ('a compact general agentic model with 3B non-embedding parameters'); Table 1 and Table 3 list total 4B / non-embedding 3B
n/a · §2.1
n/a · §3.2.1
stated · §3.2.1 ('progressively extends the maximum training context from 64K to 128K and 256K tokens'); Appendix B.1 ('Context Window: 256k tokens')
stated · §3.2.1 ('a three-stage curriculum that progressively extends the maximum training context from 64K to 128K and 256K tokens')
stated · §2.2 ('Our pre-training corpus comprises 28T tokens')
stated · §2.2
stated · §2.1, §3.2.1
stated · §3.2.1 ('Starting from the pretrained checkpoint, we conduct supervised fine-tuning with a three-stage curriculum'); turn-level loss masking via execution feedback
stated · §3.2.2 ('we first apply a two-stage RLHF procedure covering both think and non-think responses'; point-wise reward model over response quality)