Tiny Aya is a 3.35B dense decoder-only Transformer (36 layers, hidden 2048, 16 Q / 4 KV heads, SwiGLU, bias-free, parallel attention+FFN blocks) whose block is deliberately carried over from Cohere's Command A: a 3:1 interleave of sliding-window attention (4096-token window, RoPE) and full attention (NoPE). The model's real design center of gravity is the data system around that block. A single 262k-vocabulary tokenizer is shared by all five releases, trained on 50GB of Fineweb-2 with a weighting that multiplies data-distribution weight by language-family/script bucket weight so that underrepresented scripts (Khmer, Telugu, Gujarati, Ge'ez) get competitive compression. Pretraining runs 6T tokens over 70 languages plus code, with a WSD schedule and a SmolLM3-style cooldown mixture. Posttraining is region-aware: five regional SFT clusters (Europe, West Asia, South Asia, Asia-Pacific, Africa) plus a global SFT model, fed by translated and FusioNN-aggregated synthetic data from Gemma3-27B-It, Command A and DeepSeek-V3 teachers, then each region-specialized checkpoint is merged with the global model under SimMerge operator selection — producing Tiny Aya Base, Global, Earth, Fire and Water from the same weights. The report's evaluation apparatus is as much a contribution as the model: rubric-based absolute judge ratings instead of win rates, language-confusion tracking, and mean+minimum safety reporting across languages. At 8K context it targets balanced multilingual quality rather than peaks, and its edge-deployment story (q4_k_m at 2.14 GB, ~10–32 tok/s on iPhones) is the closest the atlas has to an explicitly on-device massively-multilingual model.
What the report claims is novel
Tiny Aya redefines what a small multilingual model can achieve: 3.35B parameters across 70 languages with state-of-the-art translation quality (best average on Flores and WMT24++, beating Gemma3-4B in 46/55 WMT24++ languages) and the highest multilingual safety (91.1% mean MultiJail safe rate) while drastically reducing language disparities [Abstract, §1, §5]
A balanced multilingual data mixture: tokenizer data weighted by language-family/script buckets (Abagyan et al. 2025 recipe) plus pretraining/posttraining mixtures that explicitly balance regional coverage, with prompt-level transformations (Naturalness, Cultural Adaptation, Difficulty Enhancement) to reduce translationese and English-centric framing [§2.1, §2.2, §2.3.1]
Region-aware posttraining with cluster-based SFT and SimMerge-guided checkpoint merging: region-specialized models (Earth, Fire, Water) merged with the Global model improve translation by up to +5.5 ChrF (South Asia) and +1.7 on average (Africa) while restoring global instruction-following and safety [§2.3, §3.2, §3.3, §5.2]
FusioNN (Fusion-of-NN) teacher aggregation for massively multilingual synthetic completions, with Command A as Fusor and per-language teacher selection [§2.3.1]
A comprehensive multilingual evaluation framework: rubric-based absolute ratings in lieu of win rates, language-confusion tracking, and mean-plus-minimum safety reporting to surface cross-language disparities [§4.3]
Practical edge deployment: standard llama.cpp/MLX quantization (q4_0, q4_k_m, q8_0) achieving ~10 tokens/s decode on a four-year-old iPhone 13 and 32 tokens/s on iPhone 17 Pro [§6]
Atlas assessment: Architecturally a faithful, smaller Command A: parallel blocks, 3:1 sliding-window/full-attention interleave with the same RoPE-on-SWA / NoPE-on-full split, GQA, SwiGLU, bias-free — the report states this lineage explicitly, so the block itself contributes no new mechanism to the atlas. The novelty is entirely in the data and training system: the language-family/script-bucket tokenizer weighting (from Cohere's own prior work), region-clustered SFT with SimMerge-guided merging into Earth/Fire/Water variants, FusioNN teacher aggregation, and the rubric/language-confusion evaluation methodology. vs google-gemma-3 (the closest atlas peer at similar scale): Tiny Aya trades a uniform attention block for the SWA/NoPE hybrid, and its claim to distinctiveness is balanced cross-language performance and safety rather than raw peaks — consistent with its own framing that it does not 'nominally score the highest' on discriminative tasks. The family's on-device framing (quantized MLX/llama.cpp deployment) is also more explicit than most atlas peers at this scale.
Relation to the atlas
Lineage in
Parallel Transformer blocks, 3:1 sliding-window/full attention interleave, RoPE-on-SWA / NoPE-on-full split, GQA, SwiGLU, bias-free dense layers — 'closely follows the core design choices from Command A' from Command A
Cooldown (mid-training) mixture upsampling high-quality and instruction-style data from SmolLM3-3B
Multilingual tokenizer data weighting by language-family/script buckets from Cohere Labs tokenizer work (not in atlas)
Aya family lineage (Aya 23, Aya Expanse) as the multilingual-research context for the release from Aya 23 / Aya Expanse (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
Optimizer not named (only 'optimizer states in FP32')
Pretraining batch size and peak learning-rate values not disclosed (posttraining values are given)
Tokenizer subword algorithm not named (GPT-4o-regex pre-tokenization and Fineweb-2 data described)
Normalization family (RMSNorm vs LayerNorm) and norm placement not stated
Model license not named (only the arXiv paper license CC BY-NC-ND 4.0 appears)
RoPE base frequency and per-dimension partial-RoPE not disclosed
No per-language pretraining data proportions (only posttraining cluster mixes in Appendix A, plus English shares in Table 8)
No context extension mechanism (fixed 8K per Table 2) and no statement that training ran at deployed length
No training compute (FLOPs) and no parallelism strategy disclosed
Embedding tying not disclosed
No speculative decoding or KV-cache engineering discussion
3:1 interleave of sliding-window attention (4096-token window) and full attention layers across the 36 layers, following Command A; sliding-window layers use RoPE, full-attention layers use NoPE[14]
single massively multilingual tokenizer shared by all models; 262k vocabulary trained on 50GB of Fineweb-2 sampled with a weighting that combines data-distribution weight and language-family/script bucket weight (wi = wid·wib / Σ); GPT-4o regex pre-tokenization, no normalization; competitive or superior tokens-per-character compression across scripts, especially Khmer, Telugu, Gujarati, Lao, Ge'ez[20]
large corpus of public and proprietary sources covering 70 languages plus programming-language datasets; filtering pipeline: (1) language ID and stopword filtering, (2) heuristic cleaning from raw sources, (3) deduplication, (4) domain classification and quality filtering; cooldown (mid-training) mixture upsampling the highest-quality pretraining datasets plus instruction-style datasets spanning all 70 languages[22]
Curriculum / staging
Warmup-Stable-Decay (WSD) schedule with a cooldown (mid-training) mixture following SmolLM3-3B; hyperparameters chosen via 200B-token ablation runs with 40B-token cooldowns[23]
FP8 training mixing FP8, BF16 and FP32: main weights and optimizer states in FP32, weights cast to BF16/FP8 before computation; exponentials, softmaxes, layer norms and output embeddings in FP32; attention computation in BF16[14]
Parallelism
not disclosed
Hardware
256 NVIDIA H100 GPUs for pretraining; 16 NVIDIA H100 GPUs per cluster model for posttraining (each cluster run completes within 24 hours wall-clock)[24]
GQA with 16 query heads over 4 KV heads: KV-cache footprint proportional to 4 heads (1/4 of MHA); sliding-window layers need only the 4096-token window cached
Quantisation shipped
llama.cpp formats q4_0, q4_k_m and q8_0, served via llama.cpp and MLX; Q4_K_M optimal: 2.14 GB memory, 32.4 tokens/s decode, ~1.4-point mDolly degradation (Q4_0: ~2.1 points; Q8_0: negligible)[28]
llama.cpp and MLX inference stacks for edge deployment; ~10 tokens/s decode on iPhone 13 and ~32 tokens/s on iPhone 17 Pro (Q4_K_M); low prefill throughput on older devices raises TTFT[28]
stated · arXiv stamp 'arXiv:2603.11510v1 [cs.CL] 12 Mar 2026'
stated · pipeline
stated · §1 ('a family of efficient, open-weight multilingual models'), §7 ('We release a family of 3.35B-parameter models: Tiny Aya Base, Tiny Aya Global, and region specific models Tiny Aya Water, Tiny Aya Earth, Tiny Aya Fire')
stated · §3.1 ('We pretrain Tiny Aya model for 6T tokens')
stated · §2.2
stated · §3.1, §2.2
stated · §3.1, §3.2
stated · §2.3, §3.2 (region-specific SFT on five regional data subsets plus a global SFT model over all regions; 3 epochs, cosine decay, global batch 32)
stated · §3.2 ('a minimal preference tuning phase on top of SFT for the Tiny Aya Global model... teaches the model its identity... while maintaining multilingual safety')