Laguna

poolside · analysed 2026-08-10 · skill v0.4.0

sparse-MoE hybrid text-only open-weights medium domain-specific

Design overview

Laguna XS.2 is poolside's agentic-coding MoE: 33.4B total / 3B active over 40 layers, with 256 routed experts (top-8) plus one shared expert and a sigmoid router with post-top-k score normalization. Its distinguishing feature is layer-wise attention budgeting: a 3:1 interleave of sliding-window (window 512, 64 Q-heads, RoPE base 10k) and global (48 Q-heads, RoPE base 500k, partial RoPE 50%) attention, with per-head softplus gating deciding each head's contribution. Context grows from 4K pretraining through 32K and a YaRN stage to 128K, then doubles to 256K at deployment via RoPE-base scaling without further training. Training uses Muon (Moonlight variant) under WSD with >30T tokens on 2,048 H200s; post-training is SFT (3×40B) plus CISPO, an RL stage restricted to verifiable rewards with length-weighted leave-one-out advantages. The sibling M.1 (225.8B) shares the block at larger scale.

What the report claims is novel

Layer-wise attention budgeting: per-layer attention-type allocation (3:1 SWA:GA) with per-head softplus gating and per-type Q-head counts and RoPE bases [architecture section]
CISPO reinforcement learning with verifiable rewards only [post-training section]
Long-horizon agentic coding focus with 256K deployed context [Abstract / context section]

Atlas assessment: The first per-head attention-budget design in the atlas: the SWA/GA interleave itself is familiar (Gemma 3, OLMo 3, Trinity), but allocating distinct Q-head counts and RoPE bases per attention type is a novel attention-budgeting pattern. The DeepSeekMoE-style shared+routed structure with routed-expert modulation is inherited (stated 'similar to DeepSeek-V3').

Relation to the atlas

Lineage in

Shared + routed experts with routed-expert modulation ('similar to DeepSeek-V3') from DeepSeek-V3
Muon optimizer (Moonlight variant) from Moonlight

Influence out

No descendants recorded yet.

Notable omissions

Hidden dimension and FFN inner dimension not disclosed for XS.2
No model-level Q-head count (per-layer-type allocation only)
No normalization type or activation function disclosed
No expert granularity, no training parallelism, no compute (FLOPs)
No tokenizer algorithm details beyond BPE + 100,352 vocab

Closest relatives in the atlas

Mellum 2 — 6 shared comparable fields
gpt-oss — 6 shared comparable fields
Trinity — 6 shared comparable fields

Spec sheet

Identity

Family	Laguna^[1]
Organisation	poolside^[2]
Report URL	https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf^[3]
License	Apache 2.0^[4]
Open weights	yes^[5]

Variants

Scale

Total parameters	33.4B^[6]
Active parameters / token	3B^[7]
Layers	40^[8]
KV heads	8^[9]
Head dimension	128^[10]
Vocabulary size	100352^[11]

Per-variant configs

Variant	Total	Layers	Hidden	FFN	Q heads	KV heads	Context
XS.2 (33.4B)	33.4B	40	not disclosed	not disclosed	not disclosed	8	262,144
M.1 (225.8B)	225.8B	not disclosed	not disclosed	not disclosed	not disclosed	not disclosed	not disclosed

Core block

Block type	sparse-MoE
Attention variant	hybrid^[12]
Attention layer pattern	3:1 SWA:GA interleaving (75% SWA); SWA layers window 512, 64 Q-heads, RoPE base 10k; GA layers 48 Q-heads, RoPE base 500k, partial RoPE 50%; per-head softplus gating; distinct Q-head counts and theta per attention type (layer-wise attention budgeting)^[12]
Depth mixing	sequential-residual

Expert count	256^[10]
Experts per token	8^[10]
Shared experts	1^[10]
Routing	Sigmoid router with post-top-k score normalization; routed-expert modulation (×2.5)^[12]
Load balancing	Aux loss (Qiu et al.) load balancing^[12]

Method	RoPE^[14]
Partial RoPE	yes^[15]

Attention sinks	no
Softcapping	no
Other

Context

Trained context length	131,072^[16]
Deployed context length	262,144^[17]
Extension method	YaRN on GA layers to 128K; RoPE base doubling to 256K (no training)^[13]

Tokenizer

Algorithm	BPE^[11]
Notes	100,352-token vocabulary^[11]

Training

Training tokens	30T^[18]
Curriculum / staging	none disclosed (pretraining); mid-train ~60B + SFT 3×40B + RL in post-training^[19]
Optimizer	Muon (Moonlight variant)^[19]
LR schedule	WSD schedule, peak LR 5e-4^[19]
Precision	BF16 with FP32 master weights^[19]
Hardware	2,048 H200 GPUs (M.1: 6,144)^[19]

Post-training

SFT	yes^[20]
Preference optimisation	other^[21]
Reasoning training	RL with verifiable rewards (CISPO); agentic coding emphasis^[21]
Distillation	none disclosed^[3]

Modality

Type	text-only

Inference efficiency

KV-cache design	SWA (window 512) on 75% of layers shrinks cache vs full attention; GA layers keep partial RoPE with 48 Q-heads; no latent-KV compression disclosed
Quantisation shipped	QAD (quantization-aware distillation) and SpinQuant^[8]
Speculative decoding	none disclosed^[3]
Serving optimisations	none disclosed^[3]

Evaluation

Benchmarks (report's own numbers only)

Benchmark	Value	Ref
SWE-bench Verified	69.9	eval tables
SWE-bench Multilingual	57.7	eval tables
SWE-Bench Pro	46.3	eval tables
Terminal-Bench 2.0	35.7	eval tables
BBH	80.9	eval tables
MMLU-STEM	78.1	eval tables
GSM8K	84.2	eval tables
LiveCodeBench v6	29.3	eval tables

Provenance references

stated · Title (Laguna M1 XS2 technical report)
stated · Title page byline
stated · pipeline
stated · license section (stated twice)
stated · license/release section
stated · config tables (XS.2)
stated · config tables (XS.2 3B active)
stated · quantization section
stated · config tables (GQA 8 KV heads)
stated · config tables
stated · tokenizer section
stated · architecture section
stated · context section
stated · architecture section (two bases, per attention type)
stated · architecture section (50% of dims on GA layers)
stated · context section (YaRN extension trained to 128K)
stated · context section (RoPE-scale doubling to 256K without training)
stated · training section (>30T pre-training tokens)
stated · training section
stated · post-training section (SFT, 3×40B)
stated · post-training section