LongCat-Flash-Lite is Meituan's demonstration that parameter budget can be moved out of the MoE layers entirely. The 68.5B-parameter model spends 31.4B of them — 46% of the total — on an N-gram Embedding module: each token's embedding is the average of a base 128K-vocab table plus hashed n-gram sub-tables (polynomial rolling hash, K sub-tables with linear projections back into the model space), so capacity grows with vocabulary rather than compute. Because the extra parameters are looked up in O(1) rather than routed, the model keeps total size while cutting activated parameters, which the authors show pays off most at high sparsity: their scaling study (280M/790M/1.3B activated probes over 300B tokens) finds N-gram Embedding beats parameter-equivalent expert scaling once the total/active ratio exceeds ~20, with the crossover shifting higher in wider models and shrinking in deeper ones. On top of the Longcat-Flash MoE backbone (14 shortcut layers, each with two sub-layers; 256 FFN experts plus 128 zero-experts, top-12), the report adds Embedding Amplification — scaling the embedding output by sqrt(D) before the residual merge so the first attention module does not drown it out. Context reaches 256K via YaRN at the 32K stage and a 128K mid-training phase. Inference leans on the reduced activation: Eagle3 3-step speculative decoding, wide expert parallelism, heavy kernel fusion (including Q-Norm + KV-Norm and a unified router kernel), a specialized N-gram Cache, and PDL overlap. The report defers the attention mechanism itself to the Longcat-Flash technical report, so the sequence-mixing design is not documented here.
What the report claims is novel
Embedding scaling as an orthogonal sparsity dimension: N-gram Embedding achieves a superior Pareto frontier vs expert scaling in high-sparsity regimes (crossover near total/active ratio ~20; wider models sustain the advantage to ratios beyond 50) [§1, §3.1, §3.3]
Systematic characterization of the architectural factors governing embedding-scaling efficacy: integration timing, parameter budgeting, hash-collision mitigation via vocabulary sizing, n-gram order/sub-table hyperparameters, Embedding Amplification, and width/depth interplay [§3]
LongCat-Flash-Lite: 68.5B model with 31.4B N-gram Embedding parameters (46% of total) and dynamic 2.9-4.5B activation via 128 zero-experts; outperforms the parameter-equivalent MoE baseline and is competitive with Qwen3-Next-80B-A3B, Kimi-Linear-48B-A3B and Gemini 2.5 Flash-Lite, especially in agentic tool use and coding [§6]
Inference efficiency: N-gram Cache and synchronized kernels eliminate embedding-lookup overhead; N-gram Embedding reduces MoE memory-I/O in decoding; synergy with speculative decoding (Eagle3, 3-step) via draft-side conventional embeddings and embedding caching [§4, §6.4]
Atlas assessment: The genuinely novel axis here is parameter allocation: instead of more experts or a new mixing layer, LongCat-Flash-Lite pushes 46% of its budget into a hashed n-gram embedding table (over 30B parameters), a mechanism with O(1) lookup that never touches compute. The N-gram Embedding mechanism itself is borrowed (Clark et al. 2022 / Huang et al. 2025 Over-Encoding), and the report's contribution is the scaling analysis plus system engineering. vs the atlas: the closest relative is moonshot-kimi-linear-48b-a3b (48B-A3B MoE, long context, agentic-leaning), but that model scales via linear attention while LongCat-Flash-Lite scales via embeddings and dynamic zero-expert activation — a fundamentally different sparsity strategy; the 256K YaRN extension and 128k mid-training confirm the LongCat long-context family angle. The 'superior Pareto frontier' claim rests on in-house 300B-token experiments with intersection points that shift with width — honest but not independently verifiable. Weaknesses: the attention mechanism, hidden dim, heads, optimizer, and training precision are all deferred or undisclosed, so the architecture is only partially documented in this report.
Relation to the atlas
Lineage in
N-gram Embedding (Over-Encoding): vocabulary-free n-gram tables with K sub-tables and linear projections from Over-Tokenized Transformer / CANINE (not in atlas)
Embedding Amplification (embedding output scaling / LayerNorm before residual merge) from Takase et al. 2025 (not in atlas)
YaRN context extension from YaRN (not in atlas; used by Qwen et al.)
Eagle3 speculative decoding from EAGLE-3 (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
Attention mechanism entirely deferred to the Longcat-Flash technical report (arXiv 2509.01322) — attention variant, heads, layer pattern, and KV-cache design are undisclosed here
Hidden dimension and all FFN/expert dimensions not disclosed
Optimizer, learning-rate schedule, and training precision not disclosed
Training hardware and compute (FLOPs/GPU-hours) not disclosed
Load-balancing mechanism for the 256+128 expert pools not described
N-gram hyperparameters for the final model (N, K, n-gram vocabulary size) not given (only 30x-33x base-vocab guidance from ablations)
No license named
Embedding tying not disclosed
Evaluation table values for Gemini 2.5 Flash-Lite and some rows are sourced from public reports (marked with *), not measured in-house
No RL/post-SFT pipeline at all (SFT only), unlike peers of this scale
YaRN implemented during the 32k sequence-length training stage, enabling up to 256k; training curriculum: 8k pre-training (11T) -> 128k mid-training (1.5T) -> SFT[17]
Tokenizer
Algorithm
not disclosed
Notes
not disclosed
Training
Training tokens
12.5T
Data composition
same data recipe as LongCat-Flash-Chat (Meituan, 2025); not detailed in this report[17]
Curriculum / staging
three stages: pre-training on 11T tokens at 8k sequence length, mid-training on 1.5T tokens with sequence length extended to 128k, then supervised fine-tuning[17]
Optimizer
not disclosed
LR schedule
not disclosed
Batch schedule
batch size increased at 420B tokens (Figure 10 caption: 'The loss drop at 420B tokens coincides with the batch size increases')[18]
Precision
not disclosed
Parallelism
wide EP (Expert Parallel) and SBO (Single Batch Overlap) adopted for inference deployment[10]
quantized deployment with activation quantization fused into operators; quantization format not named[10]
Speculative decoding
Eagle3 with a 3-step speculative decoding strategy; draft model uses a conventional embedding layer (no n-gram lookup) and n-gram embeddings are cached during drafting to avoid redundant verification-time computation[23]
Serving optimisations
wide EP + SBO; extensive kernel fusion (AllReduce+Residual Add+RMSNorm; AllGather+Q-Norm+KV-Norm; ReduceScatter+RMSNorm+Hidden State Combine; router Softmax+TopK+scaling+zero-expert selection); optimized splitkv-and-combine attention kernel (-50% combine latency); PDL (Programmatic Dependent Launch); N-gram Cache with custom CUDA kernels[23]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
Tau2-Airline (avg@8)
58.0
Table 2
Tau2-Retail (avg@8)
73.1
Table 2
Tau2-Telecom (avg@8)
72.8
Table 2
VitaBench (avg@4)
7.0
Table 2
SWE-Bench (acc)
54.4
Table 2
TerminalBench (acc)
33.75
Table 2
SWE-Bench Multilingual
38.1
Table 2
PRDBench
39.63
Table 2
GPQA-Diamond (avg@16)
66.78
Table 2
MMLU (acc)
85.52
Table 2
MMLU-Pro (acc)
78.29
Table 2
CEval (acc)
86.55
Table 2
CMMLU (acc)
82.48
Table 2
MATH500 (acc)
96.8
Table 2
AIME24 (avg@32)
72.19
Table 2
AIME25 (avg@32)
63.23
Table 2
BBH (base, @1.3T)
43.67
Table 1
DROP (base, @1.3T)
52.43
Table 1
GSM8K (base, @1.3T)
50.5
Table 1
BigCodeBench (base, @1.3T)
36.05
Table 1
Provenance references
stated · §6.1 ('LongCat-Flash-Lite adopts the same architecture as Longcat-Flash (Meituan, 2025)'); byline 'Meituan LongCat Team'
stated · Byline
stated · pipeline
stated · Abstract ('we introduce and open-source LongCat-Flash-Lite'); HF link huggingface.co/meituan-longcat/LongCat-Flash-Lite
stated · Abstract, §6.1
stated · §3.2.2 ('n-gram vocabulary size set to 30x the base vocabulary (128k)')
stated · §5.1 Eq. 4 / §5.2 Eq. 5 (FFN output Wd(SiLU(Wg x) ⊙ e) — SiLU-gated FFN in the PLE/PLNE formulation of the model's MLP)
stated · §6.1 ('the MoE module consists of 256 FFN experts and 128 zero-experts')
stated · §6.1 ('each token selects 12 experts')
stated · §6.4
stated · §6.1 ('we implement YARN (Peng et al., 2023) during the 32k sequence length training stage, enabling LongCat-Flash-Lite to handle sequences up to 256k tokens')
stated · §6.1 ('handle sequences up to 256k tokens')
stated · §3.3.2 ('For pre-normalization architectures, the contribution of N-gram Embedding through the identity connection (residual branch) inherently diminishes as network depth increases')