Spaces:
Sleeping
Sleeping
| { | |
| "doc_name": "DeepSeek_V4.pdf", | |
| "doc_description": "A comprehensive technical report detailing the DeepSeek-V4 series of ultra-long-context Mixture-of-Experts language models, covering their novel hybrid attention architecture, advanced training optimizations including the Muon optimizer, extensive pre-training on 32+ trillion tokens, scalable infrastructure for efficient large-context handling, rigorous multi-domain evaluations, and innovative post-training methods like on-policy distillation to achieve state-of-the-art performance and practical deployment at million-token context scales.", | |
| "structure": [ | |
| { | |
| "title": "Preface", | |
| "node_id": "0000", | |
| "start_index": 1, | |
| "end_index": 3, | |
| "summary": "The partial document presents an overview of DeepSeek-V4, a new series of highly efficient Mixture-of-Experts (MoE) language models designed to handle extremely long contexts of up to one million tokens. It introduces two main models, DeepSeek-V4-Pro and DeepSeek-V4-Flash, highlighting their large parameter sizes and activated parameters. Key architectural innovations include a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) for improved long-context efficiency, Manifold-Constrained Hyper-Connections (mHC) to enhance residual connections, and the Muon optimizer for faster and more stable training. The document details the extensive pre-training on over 32 trillion tokens and a comprehensive post-training pipeline that further boosts model capabilities. It emphasizes the models\u2019 superior efficiency in inference, requiring significantly fewer FLOPs and smaller KV cache compared to previous versions, enabling practical use of million-token contexts. The document also outlines the structure of the paper, covering architecture design, training infrastructure, pre-training and post-training methodologies, evaluation benchmarks, real-world task performance, and future directions.", | |
| "text": "DeepSeek-V4:\nTowards Highly Efficient Million-Token Context Intelligence\nDeepSeek-AI\nresearch@deepseek.com\nAbstract\nWe present a preview version of DeepSeek-V4 series, including two strong Mixture-of-\nExperts (MoE) language models \u2014 DeepSeek-V4-Pro with 1.6T parameters (49B activated) and\nDeepSeek-V4-Flash with 284B parameters (13B activated) \u2014 both supporting a context length of\none million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and op-\ntimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA)\nand Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold-\nConstrained Hyper-Connections (mHC) that enhance conventional residual connections; (3)\nand the Muon optimizer for faster convergence and greater training stability. We pre-train\nboth models on more than 32T diverse and high-quality tokens, followed by a comprehensive\npost-training pipeline that unlocks and further enhances their capabilities. DeepSeek-V4-Pro-\nMax, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for\nopen models, outperforming its predecessors in core tasks. Meanwhile, DeepSeek-V4 series are\nhighly efficient in long-context scenarios. In the one-million-token context setting, DeepSeek-\nV4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared\nwith DeepSeek-V3.2. This enables us to routinely support one-million-token contexts, thereby\nmaking long-horizon tasks and further test-time scaling more feasible. The model checkpoints\nare available athttps://huggingface.co/collections/deepseek-ai/deepseek-v4.\nSimpleQA\nVerified\n(Pass@1)HLE\n(Pass@1)Apex\nShortlist\n(Pass@1)Codeforces\n(Rating)SWE\nVerified\n(Resolved)Terminal\nBench 2.0\n(Acc)Toolathlon\n(Pass@1)020406080100Accuracy / Pass@1 (%)57.9\n46.245.375.6\n37.740.039.844.490.2\n85.9\n78.189.132063168\n3052\n80.6 80.880.6\n67.9\n65.475.1\n68.5\n51.8\n47.254.6\n48.8\nKnowledge & Reasoning Agentic CapabilitiesDeepSeek-V4-Pro-Max Claude-Opus-4.6-Max GPT-5.4-xHigh Gemini-3.1-Pro-High\n0 256 512 768 1024\nToken Position (K)0.00.20.40.60.81.01.2Single-Token FLOPs (T)\n3.7\u00d7 lower\n9.8\u00d7 lowerDeepSeek-V3.2\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\n0 256 512 768 1024\nSequence Length (K)01020304050Accumulated KV Cache (GB)\n9.5\u00d7 smaller\n13.7\u00d7 smallerDeepSeek-V3.2\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nFigure 1|Left: benchmark performance of DeepSeek-V4-Pro-Max and its counterparts.Right:\ninference FLOPs and KV cache size of DeepSeek-V4 series and DeepSeek-V3.2.Contents\n1 Introduction 4\n2 Architecture 6\n2.1 Designs Inherited from DeepSeek-V3 . . . . . . . . . . . . . . . . . . . . . . . . . . 7\n2.2 Manifold-Constrained Hyper-Connections . . . . . . . . . . . . . . . . . . . . . . 7\n2.3 Hybrid Attention with CSA and HCA . . . . . . . . . . . . . . . . . . . . . . . . . 9\n2.3.1 Compressed Sparse Attention . . . . . . . . . . . . . . . . . . . . . . . . . . 9\n2.3.2 Heavily Compressed Attention . . . . . . . . . . . . . . . . . . . . . . . . . 11\n2.3.3 Other Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12\n2.3.4 Efficiency Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13\n2.4 Muon Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14\n3 General Infrastructures 15\n3.1 Fine-Grained Communication-Computation Overlap in Expert Parallelism . . . . 15\n3.2 Flexible and Efficient Kernel Development with TileLang . . . . . . . . . . . . . . 16\n3.3 High-Performance Batch-Invariant and Deterministic Kernel Libraries . . . . . . 18\n3.4 Training Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19\n3.4.1 Efficient Implementation of Muon . . . . . . . . . . . . . . . . . . . . . . . 19\n3.4.2 Cost-Effective and Memory-Efficient Implementation ofmHC . . . . . . . 20\n3.4.3 Contextual Parallelism for Long-Context Attention . . . . . . . . . . . . . 20\n3.4.4 Extended Automatic Differentiation for Flexible Activation Checkpointing 21\n3.5 Inference Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21\n3.5.1 KV Cache Structure and Management . . . . . . . . . . . . . . . . . . . . . 21\n3.5.2 On-Disk KV Cache Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 23\n4 Pre-Training 24\n4.1 Data Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24\n4.2 Pre-Training Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24\n4.2.1 Model Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24\n4.2.2 Training Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25\n4.2.3 Mitigating Training Instability . . . . . . . . . . . . . . . . . . . . . . . . . 26\n4.3 Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27\n4.3.1 Evaluation Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27\n4.3.2 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27\n25 Post-Training 28\n5.1 Post-Training Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28\n5.1.1 Specialist Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28\n5.1.2 On-Policy Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32\n5.2 Post-Training Infrastructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33\n5.2.1 FP4 Quantization-Aware Training . . . . . . . . . . . . . . . . . . . . . . . 33\n5.2.2 Efficient Teacher Scheduling for Full-Vocabulary OPD . . . . . . . . . . . 34\n5.2.3 Preemptible and Fault-Tolerant Rollout Service . . . . . . . . . . . . . . . 34\n5.2.4 Scaling RL Framework for Million-Token Context . . . . . . . . . . . . . . 35\n5.2.5 Sandbox Infrastructure for Agentic AI . . . . . . . . . . . . . . . . . . . . . 35\n5.3 Standard Benchmark Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36\n5.3.1 Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36\n5.3.2 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37\n5.4 Performance on Real-World Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 41\n5.4.1 Chinese Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41\n5.4.2 Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42\n5.4.3 White-Collar Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42\n5.4.4 Code Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44\n6 Conclusion, Limitations, and Future Directions 44\nA Author List and Acknowledgment 54\nA.1 Author List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54\nA.2 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55\nB Evaluation Details 55\n3" | |
| }, | |
| { | |
| "title": "Introduction", | |
| "node_id": "0001", | |
| "start_index": 4, | |
| "end_index": 6, | |
| "summary": "The partial document primarily covers the introduction and architectural overview of the DeepSeek-V4 series of Large Language Models (LLMs). It highlights the challenges posed by the quadratic computational complexity of traditional attention mechanisms in handling ultra-long contexts, which limits test-time scaling and long-horizon task performance. To address this, the DeepSeek-V4 series introduces significant architectural innovations, including a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), Manifold-Constrained Hyper-Connections (mHC) to enhance residual connections, and the Muon optimizer for improved training efficiency and stability. The document details infrastructure optimizations such as fused kernels for Mixture of Experts (MoE) modules, a domain-specific language (TileLang) for development efficiency, and advanced training and inference frameworks supporting large context windows up to one million tokens. It also describes a two-stage post-training pipeline involving domain-specific expert cultivation followed by unified model consolidation via on-policy distillation. Evaluation results demonstrate that DeepSeek-V4 models achieve substantial improvements in computational efficiency, long-context handling, reasoning, knowledge, and agent tasks compared to previous versions and competitive open-source and proprietary models. The architecture section begins by noting the retention of the Transformer backbone and Multi-Token Prediction modules, with key upgrades like mHC to strengthen model capabilities.", | |
| "text": "1. Introduction\nThe emergence of reasoning models (DeepSeek-AI, 2025; OpenAI, 2024c) has established a\nnew paradigm of test-time scaling, driving substantial performance gains for Large Language\nModels (LLMs). However, this scaling paradigm is fundamentally constrained by the quadratic\ncomputational complexity of the vanilla attention mechanism (Vaswani et al., 2017), which\ncreates a prohibitive bottleneck for ultra-long contexts and reasoning processes. Concurrently,\nthe emergence of long-horizon scenarios and tasks \u2014 from complex agentic workflows to\nmassive cross-document analysis \u2014 has also made efficient support for ultra-long contexts\ncritical for future progress. While recent open-source efforts (Bai et al., 2025a; DeepSeek-AI,\n2024; MiniMax, 2025; Qwen, 2025) have advanced general capabilities, this core architectural\ninefficiency in handling ultra-long sequences remains a key impediment, limiting further gains\nfrom test-time scaling and hindering further exploration into long-horizon scenarios and tasks.\nIn order to break the efficiency barrier in ultra-long contexts, we develop the DeepSeek-V4\nseries, including the preview versions of DeepSeek-V4-Pro with 1.6T parameters (49B activated)\nand DeepSeek-V4-Flash with 284B parameters (13B activated). Through architectural innova-\ntions, DeepSeek-V4 series achieve a dramatic leap in computational efficiency for processing\nultra-long sequences. This breakthrough enables efficient support for a context length of one\nmillion tokens, ushering in a new era of million-length contexts for next-generation LLMs. We\nbelieve our capability to efficiently handle ultra-long sequences unlocks the next frontier of\ntest-time scaling, paves the way for deeper research into long-horizon tasks, and establishes a\nnecessary foundation for exploring future paradigms like online learning.\nCompared with the DeepSeek-V3 architecture (DeepSeek-AI, 2024), DeepSeek-V4 series\nretain the DeepSeekMoE framework (Dai et al., 2024) and Multi-Token Prediction (MTP) strategy,\nwhile introducing several key innovations in architecture and optimization. To enhance long-\ncontext efficiency, we design a hybrid attention mechanism combining Compressed Sparse\nAttention (CSA) and Heavily Compressed Attention (HCA). CSA compresses the KV caches\nalong the sequence dimension and then performs DeepSeek Sparse Attention (DSA) (DeepSeek-\nAI, 2025), whereas HCA applies more aggressive compression to the KV caches but keeps\ndense attention. To strengthen modeling capability, we incorporate Manifold-Constrained\nHyper-Connections (mHC) (Xie et al., 2026) that upgrade conventional residual connections.\nAdditionally, we introduce the Muon (Jordan et al., 2024; Liu et al., 2025) optimizer to the\ntraining of DeepSeek-V4 series, leading to faster convergence and improved training stability.\nTo enable efficient training and inference for DeepSeek-V4 series as well as productive de-\nvelopment, we introduce several infrastructure optimizations. First, we design and implement\na single fused kernel for MoE modules that fully overlaps computation, communication, and\nmemory access. Second, we employ TileLang (Wang et al., 2026), a Domain-Specific Language\n(DSL) to balance development productivity and runtime efficiency. Third, we provide efficient\nbatch-invariant and deterministic kernel libraries to ensure bitwise reproducibility across train-\ning and inference. Fourth, for the training framework, we extend the autograd framework with\ntensor-level checkpointing for fine-grained recomputation control; and we enhance training\nefficiency with a hybrid ZeRO strategy for the Muon optimizer, cost-effectivemHC implemen-\ntations via recomputation and fused kernels, and two-stage contextual parallelism to manage\ncompressed attention. Fifth, for the inference framework, we design a heterogeneous KV cache\nstructure with on-disk storage strategies to enable efficient shared-prefix reuse. In addition,\nduring the post-training stage, we incorporate FP4 quantization-aware training for MoE expert\nweights and the indexer QK path to reduce memory and computation.\n4By employing hybrid CSA and HCA, along with precision optimizations on computation\nand storage, DeepSeek-V4 series achieve significantly lower inference FLOPs and a substantially\nreduced KV cache size compared with DeepSeek-V3.2, especially in long-context settings. The\nright part of Figure 1 demonstrates the estimated single-token inference FLOPs and accumulated\nKV cache size of DeepSeek-V3.2 and DeepSeek-V4 series. In the scenario of 1M-token context,\neven DeepSeek-V4-Pro, which has a larger number of activated parameters, attains only 27%\nof the single-token FLOPs (measured in equivalent FP8 FLOPs) and 10% of the KV cache\nsize relative to DeepSeek-V3.2. Furthermore, DeepSeek-V4-Flash, with its smaller number of\nactivated parameters, pushes efficiency even further: in the 1M-token context setting, it achieves\nonly 10% of the single-token FLOPs and 7% of the KV cache size compared with DeepSeek-V3.2.\nAdditionally, for DeepSeek-V4 series, the routed expert parameters utilize FP4 precision. While\nthe peak FLOPs for FP4 \u00d7FP8 operations are currently the same as FP8 \u00d7FP8 on existing\nhardware, they can theoretically be implemented to be 1 /3 more efficient on future hardware,\nwhich will further enhance the efficiency of DeepSeek-V4 series.\nDuring pre-training, we train DeepSeek-V4-Flash on 32T tokens and DeepSeek-V4-Pro on 33T\ntokens, respectively. After pre-training, these two models can natively and efficiently support\n1M-length contexts. In our internal evaluations, DeepSeek-V4-Flash-Base already surpasses\nDeepSeek-V3.2-Base across a majority of benchmarks with its more parameter-efficient design.\nDeepSeek-V4-Pro-Base further extends this advantage to set a new performance standard among\nDeepSeek foundation models, achieving comprehensive superiority across reasoning, coding,\nlong-context, and world knowledge tasks.\nThe post-training pipeline of DeepSeek-V4 series features a two-stage paradigm: the inde-\npendent cultivation of domain-specific experts, followed by unified model consolidation via\non-policy distillation (Gu et al., 2024; Lu and Lab, 2025). Initially, for each target domain \u2014 such\nas mathematics, coding, agent, and instruction following \u2014 a separate expert model is trained\nindependently. The base model first undergoes Supervised Fine-Tuning (SFT) on high-quality,\ndomain-specific data to establish foundational capabilities. Subsequently, Reinforcement Learn-\ning (RL) is applied using Group Relative Policy Optimization (GRPO) (DeepSeek-AI, 2025),\nwhich further optimizes the model for domain-aligned behaviors guided by reward models\ntailored to specific success criteria. This phase yields a diverse set of specialized experts, each\nexcelling in its respective field. Finally, to integrate these distinct proficiencies, a single unified\nmodel is trained through on-policy distillation, wherein the unified model acts as the student\nlearning to optimize the reverse KL loss with teacher models.\nSummary of Core Evaluation Results\n\u2022Knowledge: In assessments of broad world knowledge, DeepSeek-V4-Pro-Max, the maxi-\nmum reasoning effort mode of DeepSeek-V4-Pro, significantly outperforms leading open-\nsource models on the SimpleQA (OpenAI, 2024d) and Chinese-SimpleQA (He et al., 2024)\nbenchmarks. Regarding educational knowledge \u2014 evaluated via MMLU-Pro (Wang et al.,\n2024b), HLE (Phan et al., 2025), and GPQA (Rein et al., 2023) \u2014 DeepSeek-V4-Pro-Max\nshows a marginal lead over its open-source counterparts. DeepSeek-V4-Pro-Max has\nsignificantly closed the gap with the leading proprietary model, Gemini-3.1-Pro, despite\nstill trailing it in these knowledge-based evaluations.\n\u2022Reasoning: Through the expansion of reasoning tokens, DeepSeek-V4-Pro-Max demon-\nstrates superior performance relative to GPT-5.2 and Gemini-3.0-Pro on standard reasoning\nbenchmarks. Nevertheless, its performance falls marginally short of GPT-5.4 and Gemini-\n3.1-Pro, suggesting a developmental trajectory that trails state-of-the-art frontier models by\napproximately 3 to 6 months. Furthermore, DeepSeek-V4-Flash-Max achieves comparable\n5Input TokensEmbeddingCSA / HCAPrediction HeadMTP Modules\nLM LossMTP Loss\nResidual Mixing\nPre-Block MixingPost -Block MixingTransformer Block \u00d7\ud835\udc3f\ud835\udc3f\nDeepSeek MoE\nResidual Mixing\nPre-Block MixingPost -Block MixingFigure 2|Overall architecture of DeepSeek-V4 series. We use hybrid CSA (Compressed Sparse\nAttention) and HCA (Heavily Compressed Attention) for attention layers, DeepSeekMoE for\nfeed-forward layers, and strengthen conventional residual connections withmHC.\nperformance to GPT-5.2 and Gemini-3.0-Pro, establishing itself as a highly cost-effective\narchitecture for complex reasoning tasks.\n\u2022Agent: On public benchmarks, DeepSeek-V4-Pro-Max is on par with leading open-source\nmodels, such as Kimi-K2.6 and GLM-5.1, but slightly worse than frontier closed models.\nIn our internal evaluation, DeepSeek-V4-Pro-Max outperforms Claude Sonnet 4.5 and\napproaches the level of Opus 4.5.\n\u2022Long-Context: DeepSeek-V4-Pro-Max delivers strong results on synthetic and real use\ncases with a 1-million-token context window, surpassing even Gemini-3.1-Pro on academic\nbenchmarks.\n\u2022DeepSeek-V4-Pro v.s. DeepSeek-V4-Flash: DeepSeek-V4-Flash-Max exhibits lower per-\nformance in knowledge evaluations due to its smaller parameter scale. However, it\nachieves comparable results on reasoning tasks when allocated a larger thinking bud-\nget. In agent evaluations, while DeepSeek-V4-Flash-Max matches the performance of\nDeepSeek-V4-Pro-Max on several benchmarks, it still trails its larger counterpart on more\ncomplex, high-difficulty tasks.\n2. Architecture\nOverall, DeepSeek-V4 series retain the Transformer (Vaswani et al., 2017) architecture and Multi-\nToken Prediction (MTP) modules (DeepSeek-AI, 2024; Gloeckle et al., 2024), while introducing\nseveral key upgrades over DeepSeek-V3: (1) firstly, we introduce the Manifold-Constrained\nHyper-Connections (mHC) (Xie et al., 2026) to strengthen conventional residual connections;\n6" | |
| }, | |
| { | |
| "title": "Architecture", | |
| "node_id": "0002", | |
| "start_index": 6, | |
| "end_index": 7, | |
| "summary": "The partial document provides a detailed overview of the DeepSeek-V4 series architecture, highlighting key innovations and design improvements over its predecessor, DeepSeek-V3. It describes the use of a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to enhance long-context efficiency, alongside the integration of Manifold-Constrained Hyper-Connections (mHC) to strengthen residual connections and improve signal stability across Transformer layers. The document also discusses the continued use of DeepSeekMoE for feed-forward networks with modifications such as a new activation function for affinity scores, removal of routing constraints, and adoption of Hash routing for expert selection. Multi-Token Prediction (MTP) modules and objectives remain consistent with the previous version. Performance evaluations indicate that DeepSeek-V4-Pro-Max achieves competitive results on public benchmarks and excels in long-context tasks, while the smaller DeepSeek-V4-Flash-Max offers comparable reasoning capabilities with some trade-offs in knowledge evaluation. The architecture retains the Transformer foundation and employs the Muon optimizer, with detailed explanations of the residual connection enhancements and MoE design choices.", | |
| "text": "Input TokensEmbeddingCSA / HCAPrediction HeadMTP Modules\nLM LossMTP Loss\nResidual Mixing\nPre-Block MixingPost -Block MixingTransformer Block \u00d7\ud835\udc3f\ud835\udc3f\nDeepSeek MoE\nResidual Mixing\nPre-Block MixingPost -Block MixingFigure 2|Overall architecture of DeepSeek-V4 series. We use hybrid CSA (Compressed Sparse\nAttention) and HCA (Heavily Compressed Attention) for attention layers, DeepSeekMoE for\nfeed-forward layers, and strengthen conventional residual connections withmHC.\nperformance to GPT-5.2 and Gemini-3.0-Pro, establishing itself as a highly cost-effective\narchitecture for complex reasoning tasks.\n\u2022Agent: On public benchmarks, DeepSeek-V4-Pro-Max is on par with leading open-source\nmodels, such as Kimi-K2.6 and GLM-5.1, but slightly worse than frontier closed models.\nIn our internal evaluation, DeepSeek-V4-Pro-Max outperforms Claude Sonnet 4.5 and\napproaches the level of Opus 4.5.\n\u2022Long-Context: DeepSeek-V4-Pro-Max delivers strong results on synthetic and real use\ncases with a 1-million-token context window, surpassing even Gemini-3.1-Pro on academic\nbenchmarks.\n\u2022DeepSeek-V4-Pro v.s. DeepSeek-V4-Flash: DeepSeek-V4-Flash-Max exhibits lower per-\nformance in knowledge evaluations due to its smaller parameter scale. However, it\nachieves comparable results on reasoning tasks when allocated a larger thinking bud-\nget. In agent evaluations, while DeepSeek-V4-Flash-Max matches the performance of\nDeepSeek-V4-Pro-Max on several benchmarks, it still trails its larger counterpart on more\ncomplex, high-difficulty tasks.\n2. Architecture\nOverall, DeepSeek-V4 series retain the Transformer (Vaswani et al., 2017) architecture and Multi-\nToken Prediction (MTP) modules (DeepSeek-AI, 2024; Gloeckle et al., 2024), while introducing\nseveral key upgrades over DeepSeek-V3: (1) firstly, we introduce the Manifold-Constrained\nHyper-Connections (mHC) (Xie et al., 2026) to strengthen conventional residual connections;\n6(2) secondly, we design a hybrid attention architecture, which greatly improves long-context\nefficiency through Compressed Sparse Attention and Heavily Compressed Attention. (3) thirdly,\nwe employ Muon (Jordan et al., 2024; Liu et al., 2025) as the optimizer. For the Mixture-of-\nExperts (MoE) components, we still adopt the DeepSeekMoE (Dai et al., 2024) architecture, with\nonly minor adjustments from DeepSeek-V3. The Multi-Token Prediction (MTP) (DeepSeek-AI,\n2024; Gloeckle et al., 2024; Li et al., 2024; Qi et al., 2020) configuration remains identical to\nthat of DeepSeek-V3. All other unspecified details follow the settings established in DeepSeek-\nV3 (DeepSeek-AI, 2024). Figure 2 illustrates the overall architecture of DeepSeek-V4, and the\ndetails are described below.\n2.1. Designs Inherited from DeepSeek-V3\nMixture-of-Experts.As previous DeepSeek-series models (DeepSeek-AI, 2024; DeepSeek-AI,\n2024), DeepSeek-V4 series also adopt the DeepSeekMoE paradigm (Dai et al., 2024) for Feed-\nForward Networks (FFNs), which sets fine-grained routed experts and shared experts. Different\nfrom DeepSeek-V3, we change the activation function that computes the affinity scores from\nSigmoid(\u00b7) into Sqrt(Softplus(\u00b7)) . For load balancing, we also employ the auxiliary-loss-free\nstrategy (DeepSeek-AI, 2024; Wang et al., 2024a), augmented by a slight sequence-wise balance\nloss that prevents extreme imbalance within individual sequences. For DeepSeek-V4, we remove\nthe constraint on the number of routing target nodes, and carefully redesign the parallelism\nstrategy to maintain training efficiency. Furthermore, compared with DeepSeek-V3, we replace\nthe dense FFN layers in the initial several Transformer blocks with MoE layers that employ\nHash routing (Roller et al., 2021). The Hash routing strategy determines the target experts of\neach token according to a predefined hash function with regard to the input token ID.\nMulti-Token Prediction.As DeepSeek-V3, DeepSeek-V4 series also set MTP modules and\nobjectives. Given that the MTP strategy has been validated in DeepSeek-V3, we adopt the same\nstrategy for DeepSeek-V4 series without modification.\n2.2. Manifold-Constrained Hyper-Connections\nAs shown in Figure 2, DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections\n(mHC) (Xie et al., 2026) to strengthen the conventional residual connections between adjacent\nTransformer blocks. Compared with naive Hyper-Connections (HC) (Zhu et al., 2025), the core\nidea ofmHC is to constrain the residual mapping onto a specific manifold, and thus enhance the\nstability of signal propagation across layers while preserving model expressivity. This subsection\nbriefly introduces the standard HC and describes how we designmHC for stable training.\nStandard Hyper-Connections.The standard HC expands the width of the residual stream\nby a factor of \ud835\udc5bhc. Specifically, the shape of the residual stream is expanded from R\ud835\udc51toR\ud835\udc5bhc\u00d7\ud835\udc51,\nwhere\ud835\udc51is the hidden size of the actual layer input. Let \ud835\udc4b\ud835\udc59=[x\ud835\udc59,1;...;x\ud835\udc59,\ud835\udc5bhc]\ud835\udc47\u2208R\ud835\udc5bhc\u00d7\ud835\udc51be the\nresidual state before the \ud835\udc59-th layer. HC introduces three linear mappings: an input mapping\n\ud835\udc34\ud835\udc59\u2208R1\u00d7\ud835\udc5b hc, a residual transformation \ud835\udc35\ud835\udc59\u2208R\ud835\udc5bhc\u00d7\ud835\udc5bhc, and an output mapping \ud835\udc36\ud835\udc59\u2208R\ud835\udc5bhc\u00d71. The\nupdate of the residual state is then formulated as:\n\ud835\udc4b\ud835\udc59+1=\ud835\udc35\ud835\udc59\ud835\udc4b\ud835\udc59+\ud835\udc36\ud835\udc59F\ud835\udc59(\ud835\udc34\ud835\udc59\ud835\udc4b\ud835\udc59), (1)\nwhereF\ud835\udc59denotes the \ud835\udc59-th layer (e.g., an MoE layer), whose input and output shapes are both\nR\ud835\udc51. Note that the actual layer input \ud835\udc34\ud835\udc59\ud835\udc4b\ud835\udc59\u2208R\ud835\udc51is also\ud835\udc51-dimensional, so the expanded residual\n7", | |
| "nodes": [ | |
| { | |
| "title": "Designs Inherited from DeepSeek-V3", | |
| "node_id": "0003", | |
| "start_index": 7, | |
| "end_index": 7, | |
| "summary": "The partial document discusses the design and improvements of the DeepSeek-V4 model architecture. Key points include the introduction of a hybrid attention mechanism that enhances long-context efficiency through Compressed Sparse Attention and Heavily Compressed Attention. The model uses the Muon optimizer and retains the DeepSeekMoE architecture for Mixture-of-Experts components with minor modifications, including a change in the activation function for affinity scores and a new load balancing strategy without auxiliary loss. The Multi-Token Prediction configuration remains unchanged from DeepSeek-V3. Additionally, DeepSeek-V4 incorporates Manifold-Constrained Hyper-Connections (mHC) to improve residual connections between Transformer blocks by constraining residual mappings onto a specific manifold, thereby enhancing training stability and signal propagation while maintaining model expressivity. The document also details the mathematical formulation of standard Hyper-Connections as a basis for the mHC design.", | |
| "text": "(2) secondly, we design a hybrid attention architecture, which greatly improves long-context\nefficiency through Compressed Sparse Attention and Heavily Compressed Attention. (3) thirdly,\nwe employ Muon (Jordan et al., 2024; Liu et al., 2025) as the optimizer. For the Mixture-of-\nExperts (MoE) components, we still adopt the DeepSeekMoE (Dai et al., 2024) architecture, with\nonly minor adjustments from DeepSeek-V3. The Multi-Token Prediction (MTP) (DeepSeek-AI,\n2024; Gloeckle et al., 2024; Li et al., 2024; Qi et al., 2020) configuration remains identical to\nthat of DeepSeek-V3. All other unspecified details follow the settings established in DeepSeek-\nV3 (DeepSeek-AI, 2024). Figure 2 illustrates the overall architecture of DeepSeek-V4, and the\ndetails are described below.\n2.1. Designs Inherited from DeepSeek-V3\nMixture-of-Experts.As previous DeepSeek-series models (DeepSeek-AI, 2024; DeepSeek-AI,\n2024), DeepSeek-V4 series also adopt the DeepSeekMoE paradigm (Dai et al., 2024) for Feed-\nForward Networks (FFNs), which sets fine-grained routed experts and shared experts. Different\nfrom DeepSeek-V3, we change the activation function that computes the affinity scores from\nSigmoid(\u00b7) into Sqrt(Softplus(\u00b7)) . For load balancing, we also employ the auxiliary-loss-free\nstrategy (DeepSeek-AI, 2024; Wang et al., 2024a), augmented by a slight sequence-wise balance\nloss that prevents extreme imbalance within individual sequences. For DeepSeek-V4, we remove\nthe constraint on the number of routing target nodes, and carefully redesign the parallelism\nstrategy to maintain training efficiency. Furthermore, compared with DeepSeek-V3, we replace\nthe dense FFN layers in the initial several Transformer blocks with MoE layers that employ\nHash routing (Roller et al., 2021). The Hash routing strategy determines the target experts of\neach token according to a predefined hash function with regard to the input token ID.\nMulti-Token Prediction.As DeepSeek-V3, DeepSeek-V4 series also set MTP modules and\nobjectives. Given that the MTP strategy has been validated in DeepSeek-V3, we adopt the same\nstrategy for DeepSeek-V4 series without modification.\n2.2. Manifold-Constrained Hyper-Connections\nAs shown in Figure 2, DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections\n(mHC) (Xie et al., 2026) to strengthen the conventional residual connections between adjacent\nTransformer blocks. Compared with naive Hyper-Connections (HC) (Zhu et al., 2025), the core\nidea ofmHC is to constrain the residual mapping onto a specific manifold, and thus enhance the\nstability of signal propagation across layers while preserving model expressivity. This subsection\nbriefly introduces the standard HC and describes how we designmHC for stable training.\nStandard Hyper-Connections.The standard HC expands the width of the residual stream\nby a factor of \ud835\udc5bhc. Specifically, the shape of the residual stream is expanded from R\ud835\udc51toR\ud835\udc5bhc\u00d7\ud835\udc51,\nwhere\ud835\udc51is the hidden size of the actual layer input. Let \ud835\udc4b\ud835\udc59=[x\ud835\udc59,1;...;x\ud835\udc59,\ud835\udc5bhc]\ud835\udc47\u2208R\ud835\udc5bhc\u00d7\ud835\udc51be the\nresidual state before the \ud835\udc59-th layer. HC introduces three linear mappings: an input mapping\n\ud835\udc34\ud835\udc59\u2208R1\u00d7\ud835\udc5b hc, a residual transformation \ud835\udc35\ud835\udc59\u2208R\ud835\udc5bhc\u00d7\ud835\udc5bhc, and an output mapping \ud835\udc36\ud835\udc59\u2208R\ud835\udc5bhc\u00d71. The\nupdate of the residual state is then formulated as:\n\ud835\udc4b\ud835\udc59+1=\ud835\udc35\ud835\udc59\ud835\udc4b\ud835\udc59+\ud835\udc36\ud835\udc59F\ud835\udc59(\ud835\udc34\ud835\udc59\ud835\udc4b\ud835\udc59), (1)\nwhereF\ud835\udc59denotes the \ud835\udc59-th layer (e.g., an MoE layer), whose input and output shapes are both\nR\ud835\udc51. Note that the actual layer input \ud835\udc34\ud835\udc59\ud835\udc4b\ud835\udc59\u2208R\ud835\udc51is also\ud835\udc51-dimensional, so the expanded residual\n7" | |
| }, | |
| { | |
| "title": "Manifold-Constrained Hyper-Connections", | |
| "node_id": "0004", | |
| "start_index": 7, | |
| "end_index": 9, | |
| "summary": "The partial document primarily covers the architectural innovations and design improvements introduced in the DeepSeek-V4 model series. Key points include:\n\n1. **Hybrid Attention Architecture:** The design of a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to significantly enhance efficiency for extremely long context lengths, enabling practical handling of up to one million tokens. CSA compresses Key-Value (KV) caches and applies sparse attention, while HCA performs more aggressive compression for further computational savings.\n\n2. **Inheritance and Modifications from DeepSeek-V3:** DeepSeek-V4 retains core components from DeepSeek-V3, such as the Mixture-of-Experts (MoE) architecture (DeepSeekMoE) and Multi-Token Prediction (MTP) strategy, with modifications including a new activation function for affinity scores, removal of routing target constraints, redesigned parallelism, and replacement of initial dense FFN layers with MoE layers using Hash routing.\n\n3. **Manifold-Constrained Hyper-Connections (mHC):** Introduction of mHC to improve residual connections between Transformer blocks by constraining residual mappings to the manifold of doubly stochastic matrices. This constraint ensures numerical stability and non-expansive transformations, addressing instability issues found in standard Hyper-Connections when stacking multiple layers.\n\n4. **Dynamic Parameterization and Stability Enhancements:** Parameters for the linear mappings in mHC are dynamically generated with input-dependent and static components, followed by the application of constraints (e.g., via Sigmoid functions and Sinkhorn-Knopp algorithm) to maintain boundedness and stability during training.\n\n5. **Use of Advanced Optimizers and Architectures:** Employment of the Muon optimizer and continuation of the DeepSeekMoE architecture with minor adjustments, alongside maintaining the MTP configuration from previous versions.\n\nOverall, the document details the technical advancements in DeepSeek-V4 that improve long-context handling, training stability, and computational efficiency through novel attention mechanisms, refined MoE routing, and stable residual connection designs.", | |
| "text": "(2) secondly, we design a hybrid attention architecture, which greatly improves long-context\nefficiency through Compressed Sparse Attention and Heavily Compressed Attention. (3) thirdly,\nwe employ Muon (Jordan et al., 2024; Liu et al., 2025) as the optimizer. For the Mixture-of-\nExperts (MoE) components, we still adopt the DeepSeekMoE (Dai et al., 2024) architecture, with\nonly minor adjustments from DeepSeek-V3. The Multi-Token Prediction (MTP) (DeepSeek-AI,\n2024; Gloeckle et al., 2024; Li et al., 2024; Qi et al., 2020) configuration remains identical to\nthat of DeepSeek-V3. All other unspecified details follow the settings established in DeepSeek-\nV3 (DeepSeek-AI, 2024). Figure 2 illustrates the overall architecture of DeepSeek-V4, and the\ndetails are described below.\n2.1. Designs Inherited from DeepSeek-V3\nMixture-of-Experts.As previous DeepSeek-series models (DeepSeek-AI, 2024; DeepSeek-AI,\n2024), DeepSeek-V4 series also adopt the DeepSeekMoE paradigm (Dai et al., 2024) for Feed-\nForward Networks (FFNs), which sets fine-grained routed experts and shared experts. Different\nfrom DeepSeek-V3, we change the activation function that computes the affinity scores from\nSigmoid(\u00b7) into Sqrt(Softplus(\u00b7)) . For load balancing, we also employ the auxiliary-loss-free\nstrategy (DeepSeek-AI, 2024; Wang et al., 2024a), augmented by a slight sequence-wise balance\nloss that prevents extreme imbalance within individual sequences. For DeepSeek-V4, we remove\nthe constraint on the number of routing target nodes, and carefully redesign the parallelism\nstrategy to maintain training efficiency. Furthermore, compared with DeepSeek-V3, we replace\nthe dense FFN layers in the initial several Transformer blocks with MoE layers that employ\nHash routing (Roller et al., 2021). The Hash routing strategy determines the target experts of\neach token according to a predefined hash function with regard to the input token ID.\nMulti-Token Prediction.As DeepSeek-V3, DeepSeek-V4 series also set MTP modules and\nobjectives. Given that the MTP strategy has been validated in DeepSeek-V3, we adopt the same\nstrategy for DeepSeek-V4 series without modification.\n2.2. Manifold-Constrained Hyper-Connections\nAs shown in Figure 2, DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections\n(mHC) (Xie et al., 2026) to strengthen the conventional residual connections between adjacent\nTransformer blocks. Compared with naive Hyper-Connections (HC) (Zhu et al., 2025), the core\nidea ofmHC is to constrain the residual mapping onto a specific manifold, and thus enhance the\nstability of signal propagation across layers while preserving model expressivity. This subsection\nbriefly introduces the standard HC and describes how we designmHC for stable training.\nStandard Hyper-Connections.The standard HC expands the width of the residual stream\nby a factor of \ud835\udc5bhc. Specifically, the shape of the residual stream is expanded from R\ud835\udc51toR\ud835\udc5bhc\u00d7\ud835\udc51,\nwhere\ud835\udc51is the hidden size of the actual layer input. Let \ud835\udc4b\ud835\udc59=[x\ud835\udc59,1;...;x\ud835\udc59,\ud835\udc5bhc]\ud835\udc47\u2208R\ud835\udc5bhc\u00d7\ud835\udc51be the\nresidual state before the \ud835\udc59-th layer. HC introduces three linear mappings: an input mapping\n\ud835\udc34\ud835\udc59\u2208R1\u00d7\ud835\udc5b hc, a residual transformation \ud835\udc35\ud835\udc59\u2208R\ud835\udc5bhc\u00d7\ud835\udc5bhc, and an output mapping \ud835\udc36\ud835\udc59\u2208R\ud835\udc5bhc\u00d71. The\nupdate of the residual state is then formulated as:\n\ud835\udc4b\ud835\udc59+1=\ud835\udc35\ud835\udc59\ud835\udc4b\ud835\udc59+\ud835\udc36\ud835\udc59F\ud835\udc59(\ud835\udc34\ud835\udc59\ud835\udc4b\ud835\udc59), (1)\nwhereF\ud835\udc59denotes the \ud835\udc59-th layer (e.g., an MoE layer), whose input and output shapes are both\nR\ud835\udc51. Note that the actual layer input \ud835\udc34\ud835\udc59\ud835\udc4b\ud835\udc59\u2208R\ud835\udc51is also\ud835\udc51-dimensional, so the expanded residual\n7width does not influence the design of the inner layers. HC decouples the residual width from\nthe actual hidden size, offering a complementary scaling axis with minimal computational\noverhead, as \ud835\udc5bhcis typically much smaller than the hidden size \ud835\udc51. However, even though HC\nhas demonstrated potential in improving model performance, we find that the training will\nfrequently exhibit numerical instability when stacking multiple layers, which hinders the scaling\nof HC.\nManifold-Constrained Residual Mapping.The core innovation ofmHC is to constrain the\nresidual mapping matrix \ud835\udc35\ud835\udc59to the manifold of doubly stochastic matrices (the Birkhoff polytope)\nM, and thus enhance the stability of signal propagation across layers:\n\ud835\udc35\ud835\udc59\u2208M\u2254{\ud835\udc40\u2208R\ud835\udc5b\u00d7\ud835\udc5b|\ud835\udc401\ud835\udc5b=1\ud835\udc5b,1\ud835\udc47\n\ud835\udc5b\ud835\udc40=1\ud835\udc47\n\ud835\udc5b,\ud835\udc40\u2a7e0}. (2)\nThis constraint ensures that the spectral norm of the mapping matrix \u2225\ud835\udc35\ud835\udc59\u22252is bounded by 1, so\nthe residual transformation is non-expansive, which increases the numerical stability during both\nthe forward pass and backpropagation. Besides, the set Mis closed under multiplication, which\nguarantees stability in the scenarios of deep stacks ofmHC. In addition, the input transformation\n\ud835\udc34\ud835\udc59and output transformation \ud835\udc36\ud835\udc59are also constrained to be non-negative and bounded via a\nSigmoid function to avoid the risk of signal cancellation.\nDynamic Parameterization.The parameters of three linear mappings are dynamically gen-\nerated, which are decomposed into a dynamic (input-dependent) component and a static\n(input-independent) component. Given the input \ud835\udc4b\ud835\udc59\u2208R\ud835\udc5bhc\u00d7\ud835\udc51, it is first flattened and normal-\nized: \u02c6\ud835\udc4b\ud835\udc59=RMSNorm(vec(\ud835\udc4b \ud835\udc59))\u2208R1\u00d7\ud835\udc5b hc\ud835\udc51. Then, we follow the conventional HC to generate the\nunconstrained raw parameters \u02dc\ud835\udc34\ud835\udc59\u2208R1\u00d7\ud835\udc5b hc,\u02dc\ud835\udc35\ud835\udc59\u2208R\ud835\udc5bhc\u00d7\ud835\udc5bhc, and \u02dc\ud835\udc36\ud835\udc59\u2208R\ud835\udc5bhc\u00d71:\n\u02dc\ud835\udc34\ud835\udc59=\ud835\udefcpre\n\ud835\udc59\u00b7(\u02c6\ud835\udc4b\ud835\udc59\ud835\udc4apre\n\ud835\udc59)+\ud835\udc46pre\n\ud835\udc59, (3)\n\u02dc\ud835\udc35\ud835\udc59=\ud835\udefcres\n\ud835\udc59\u00b7Mat( \u02c6\ud835\udc4b\ud835\udc59\ud835\udc4ares\n\ud835\udc59)+\ud835\udc46res\n\ud835\udc59, (4)\n\u02dc\ud835\udc36\ud835\udc59=\ud835\udefcpost\n\ud835\udc59\u00b7(\u02c6\ud835\udc4b\ud835\udc59\ud835\udc4apost\n\ud835\udc59)\ud835\udc47+\ud835\udc46post\n\ud835\udc59, (5)\nwhere\ud835\udc4apre\n\ud835\udc59,\ud835\udc4apost\n\ud835\udc59\u2208R\ud835\udc5bhc\ud835\udc51\u00d7\ud835\udc5b hcand\ud835\udc4ares\n\ud835\udc59\u2208R\ud835\udc5bhc\ud835\udc51\u00d7\ud835\udc5b2\nhcare learnable parameters for generating the\ndynamic components; Mat(\u00b7) reshapes a vector of size 1 \u00d7\ud835\udc5b2\nhcinto a matrix of size \ud835\udc5bhc\u00d7\ud835\udc5b hc;\n\ud835\udc46pre\n\ud835\udc59\u2208R1\u00d7\ud835\udc5b hc,\ud835\udc46post\n\ud835\udc59\u2208R\ud835\udc5bhc\u00d71, and\ud835\udc46res\n\ud835\udc59\u2208R\ud835\udc5bhc\u00d7\ud835\udc5bhcare learnable static biases; and \ud835\udefcpre\n\ud835\udc59,\ud835\udefcres\n\ud835\udc59,\ud835\udefcpost\n\ud835\udc59\u2208R\nare learnable gating factors initialized to small values.\nApplying Parameter Constraints.After obtaining the unconstrained raw parameters \u02dc\ud835\udc34\ud835\udc59,\u02dc\ud835\udc35\ud835\udc59,\u02dc\ud835\udc36\ud835\udc59,\nwe then apply constraints described earlier to them to enhance the numerical stability. To be\nspecific, for the input and output mappings, we employ a Sigmoid function \ud835\udf0e(\u00b7)to ensure their\nnon-negativity and boundedness:\n\ud835\udc34\ud835\udc59=\ud835\udf0e( \u02dc\ud835\udc34\ud835\udc59), (6)\n\ud835\udc36\ud835\udc59=2\ud835\udf0e( \u02dc\ud835\udc36\ud835\udc59). (7)\nAs for the residual mapping \u02dc\ud835\udc35\ud835\udc59, we project it onto the manifold of doubly stochastic matrices M.\nThis is achieved by the Sinkhorn-Knopp algorithm, which first applies an exponential function\nto\u02dc\ud835\udc35\ud835\udc59to ensure positivity, getting \ud835\udc40(0)=exp( \u02dc\ud835\udc35\ud835\udc59), and then iteratively performs column and row\nnormalization:\n\ud835\udc40(\ud835\udc61)=T\ud835\udc5f(T\ud835\udc50(\ud835\udc40(\ud835\udc61\u22121))), (8)\nwhereT\ud835\udc5fandT\ud835\udc50denote row and column normalization, respectively. This iteration converges to\na constrained doubly stochastic matrix\ud835\udc35 \ud835\udc59=\ud835\udc40(\ud835\udc61max). We choose\ud835\udc61 max=20 as a practical value.\n8\u2026 Hidden States of KV Tokens\u2026Compressed \nIndexer Keys\nHidden State of Query TokenMulti- Query \nAttention\nCompressed \nKV Entries\u2026Top-k\nSelectorSelected \nCompressed\nKV Entries\u2026Shared Key -Value Multi -Query Attention\nIndexer Queries QueriesSliding Window\nKV EntriesConcatenation\nToken- Level\nCompressor\u2026\nToken- Level\nCompressorLightning Indexer\nIndex ScoresFigure 3|Core architectures of CSA. It compresses the number of KV entries to1\n\ud835\udc5atimes, and\nthen applies DeepSeek Sparse Attention for further acceleration. Additionally, a small set of\nsliding window KV entries is combined with the selected compressed KV entries to enhance\nlocal fine-grained dependencies.\n2.3. Hybrid Attention with CSA and HCA\nAs the context length reaches extreme scales, the attention mechanism emerges as the dominant\ncomputational bottleneck in a model. For DeepSeek-V4, we design two efficient attention\narchitectures \u2014 Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)\n\u2014 and employ their interleaved hybrid configuration, which substantially reduces the compu-\ntational cost of attention in long-text scenarios. CSA integrates both compression and sparse\nattention strategies: it first compresses the Key-Value (KV) cache of every \ud835\udc5atokens into one\nentry, and then applies DeepSeek Sparse Attention (DSA) (DeepSeek-AI, 2025) where each query\ntoken attends to only \ud835\udc58compressed KV entries. HCA aims for extreme compression by consol-\nidating the KV cache of every \ud835\udc5a\u2032(\u226b\ud835\udc5a ) tokens into a single entry. The hybrid architecture of\nCSA and HCA remarkably improves the long-context efficiency of DeepSeek-V4 series, making\none-million-token context feasible in practice. This subsection describes the core techniques\nof our hybrid attention architecture, and we also provide an open-source implementation1to\nspecify more details unambiguously.\n2.3.1. Compressed Sparse Attention\nThe core architecture of CSA is illustrated in Figure 3, which first compresses the KV cache of each\n\ud835\udc5atokens into one entry, and then applies DeepSeek Sparse Attention for further acceleration.\nCompressed Key-Value Entries.Let \ud835\udc3b\u2208R\ud835\udc5b\u00d7\ud835\udc51be a sequence of input hidden states, where\n\ud835\udc5bis the sequence length and \ud835\udc51is the hidden size. CSA first computes two series of KV entries\n\ud835\udc36\ud835\udc4e,\ud835\udc36\ud835\udc4f\u2208R\ud835\udc5b\u00d7\ud835\udc50and their corresponding compression weights \ud835\udc4d\ud835\udc4e,\ud835\udc4d\ud835\udc4f\u2208R\ud835\udc5b\u00d7\ud835\udc50, where\ud835\udc50is the head\n1https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference\n9" | |
| }, | |
| { | |
| "title": "Hybrid Attention with CSA and HCA", | |
| "node_id": "0005", | |
| "start_index": 9, | |
| "end_index": 9, | |
| "summary": "The partial document discusses advanced attention mechanisms designed to efficiently handle extremely long context lengths in language models, specifically within the DeepSeek-V4 architecture. It introduces two key attention architectures: Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA reduces computational complexity by compressing the Key-Value (KV) cache of every m tokens into a single entry and then applying sparse attention where each query attends to only a subset of these compressed entries. HCA further compresses the KV cache by consolidating a much larger number of tokens (m') into one entry for extreme compression. The document highlights a hybrid approach that interleaves CSA and HCA to significantly improve efficiency, enabling practical handling of contexts up to one million tokens. It also describes the core components of CSA, including token-level compression, sparse attention application, and the integration of sliding window KV entries to preserve local dependencies. An open-source implementation is referenced for further technical details.", | |
| "text": "\u2026 Hidden States of KV Tokens\u2026Compressed \nIndexer Keys\nHidden State of Query TokenMulti- Query \nAttention\nCompressed \nKV Entries\u2026Top-k\nSelectorSelected \nCompressed\nKV Entries\u2026Shared Key -Value Multi -Query Attention\nIndexer Queries QueriesSliding Window\nKV EntriesConcatenation\nToken- Level\nCompressor\u2026\nToken- Level\nCompressorLightning Indexer\nIndex ScoresFigure 3|Core architectures of CSA. It compresses the number of KV entries to1\n\ud835\udc5atimes, and\nthen applies DeepSeek Sparse Attention for further acceleration. Additionally, a small set of\nsliding window KV entries is combined with the selected compressed KV entries to enhance\nlocal fine-grained dependencies.\n2.3. Hybrid Attention with CSA and HCA\nAs the context length reaches extreme scales, the attention mechanism emerges as the dominant\ncomputational bottleneck in a model. For DeepSeek-V4, we design two efficient attention\narchitectures \u2014 Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)\n\u2014 and employ their interleaved hybrid configuration, which substantially reduces the compu-\ntational cost of attention in long-text scenarios. CSA integrates both compression and sparse\nattention strategies: it first compresses the Key-Value (KV) cache of every \ud835\udc5atokens into one\nentry, and then applies DeepSeek Sparse Attention (DSA) (DeepSeek-AI, 2025) where each query\ntoken attends to only \ud835\udc58compressed KV entries. HCA aims for extreme compression by consol-\nidating the KV cache of every \ud835\udc5a\u2032(\u226b\ud835\udc5a ) tokens into a single entry. The hybrid architecture of\nCSA and HCA remarkably improves the long-context efficiency of DeepSeek-V4 series, making\none-million-token context feasible in practice. This subsection describes the core techniques\nof our hybrid attention architecture, and we also provide an open-source implementation1to\nspecify more details unambiguously.\n2.3.1. Compressed Sparse Attention\nThe core architecture of CSA is illustrated in Figure 3, which first compresses the KV cache of each\n\ud835\udc5atokens into one entry, and then applies DeepSeek Sparse Attention for further acceleration.\nCompressed Key-Value Entries.Let \ud835\udc3b\u2208R\ud835\udc5b\u00d7\ud835\udc51be a sequence of input hidden states, where\n\ud835\udc5bis the sequence length and \ud835\udc51is the hidden size. CSA first computes two series of KV entries\n\ud835\udc36\ud835\udc4e,\ud835\udc36\ud835\udc4f\u2208R\ud835\udc5b\u00d7\ud835\udc50and their corresponding compression weights \ud835\udc4d\ud835\udc4e,\ud835\udc4d\ud835\udc4f\u2208R\ud835\udc5b\u00d7\ud835\udc50, where\ud835\udc50is the head\n1https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference\n9", | |
| "nodes": [ | |
| { | |
| "title": "Compressed Sparse Attention", | |
| "node_id": "0006", | |
| "start_index": 9, | |
| "end_index": 11, | |
| "summary": "The partial document focuses on advanced attention mechanisms designed to efficiently handle extremely long context lengths in language models, specifically within the DeepSeek-V4 architecture. It introduces two key attention architectures: Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), which are combined in a hybrid configuration to significantly reduce computational costs for processing long sequences, enabling contexts up to one million tokens.\n\nCSA compresses the Key-Value (KV) cache by aggregating every m tokens into a single compressed entry, then applies DeepSeek Sparse Attention to attend selectively to top-k compressed KV entries using a Lightning Indexer for efficient sparse selection. It employs multi-query attention (MQA) on these selected entries and uses a grouped output projection strategy to reduce computational overhead in projecting attention outputs.\n\nHCA performs more aggressive compression by consolidating every m' (much larger than m) tokens into single KV entries without sparse attention, and incorporates a small sliding window of KV entries to preserve local fine-grained dependencies.\n\nThe document details the mathematical formulation of compression, indexing, and attention operations, including trainable parameters, compression weights, and projection matrices. It also highlights the architectural design choices that balance compression and attention sparsity to optimize performance and scalability in long-text scenarios. An open-source implementation is referenced for further technical specifics.", | |
| "text": "\u2026 Hidden States of KV Tokens\u2026Compressed \nIndexer Keys\nHidden State of Query TokenMulti- Query \nAttention\nCompressed \nKV Entries\u2026Top-k\nSelectorSelected \nCompressed\nKV Entries\u2026Shared Key -Value Multi -Query Attention\nIndexer Queries QueriesSliding Window\nKV EntriesConcatenation\nToken- Level\nCompressor\u2026\nToken- Level\nCompressorLightning Indexer\nIndex ScoresFigure 3|Core architectures of CSA. It compresses the number of KV entries to1\n\ud835\udc5atimes, and\nthen applies DeepSeek Sparse Attention for further acceleration. Additionally, a small set of\nsliding window KV entries is combined with the selected compressed KV entries to enhance\nlocal fine-grained dependencies.\n2.3. Hybrid Attention with CSA and HCA\nAs the context length reaches extreme scales, the attention mechanism emerges as the dominant\ncomputational bottleneck in a model. For DeepSeek-V4, we design two efficient attention\narchitectures \u2014 Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)\n\u2014 and employ their interleaved hybrid configuration, which substantially reduces the compu-\ntational cost of attention in long-text scenarios. CSA integrates both compression and sparse\nattention strategies: it first compresses the Key-Value (KV) cache of every \ud835\udc5atokens into one\nentry, and then applies DeepSeek Sparse Attention (DSA) (DeepSeek-AI, 2025) where each query\ntoken attends to only \ud835\udc58compressed KV entries. HCA aims for extreme compression by consol-\nidating the KV cache of every \ud835\udc5a\u2032(\u226b\ud835\udc5a ) tokens into a single entry. The hybrid architecture of\nCSA and HCA remarkably improves the long-context efficiency of DeepSeek-V4 series, making\none-million-token context feasible in practice. This subsection describes the core techniques\nof our hybrid attention architecture, and we also provide an open-source implementation1to\nspecify more details unambiguously.\n2.3.1. Compressed Sparse Attention\nThe core architecture of CSA is illustrated in Figure 3, which first compresses the KV cache of each\n\ud835\udc5atokens into one entry, and then applies DeepSeek Sparse Attention for further acceleration.\nCompressed Key-Value Entries.Let \ud835\udc3b\u2208R\ud835\udc5b\u00d7\ud835\udc51be a sequence of input hidden states, where\n\ud835\udc5bis the sequence length and \ud835\udc51is the hidden size. CSA first computes two series of KV entries\n\ud835\udc36\ud835\udc4e,\ud835\udc36\ud835\udc4f\u2208R\ud835\udc5b\u00d7\ud835\udc50and their corresponding compression weights \ud835\udc4d\ud835\udc4e,\ud835\udc4d\ud835\udc4f\u2208R\ud835\udc5b\u00d7\ud835\udc50, where\ud835\udc50is the head\n1https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference\n9dimension:\n\ud835\udc36\ud835\udc4e=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc4e\ud835\udc3e\ud835\udc49,\ud835\udc36\ud835\udc4f=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc4f\ud835\udc3e\ud835\udc49, (9)\n\ud835\udc4d\ud835\udc4e=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc4e\ud835\udc4d,\ud835\udc4d\ud835\udc4f=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc4f\ud835\udc4d, (10)\nwhere\ud835\udc4a\ud835\udc4e\ud835\udc3e\ud835\udc49,\ud835\udc4a\ud835\udc4f\ud835\udc3e\ud835\udc49,\ud835\udc4a\ud835\udc4e\ud835\udc4d,\ud835\udc4a\ud835\udc4f\ud835\udc4d\u2208R\ud835\udc51\u00d7\ud835\udc50are trainable parameters. Next, each \ud835\udc5aKV entries in \ud835\udc36\ud835\udc4eand\n\ud835\udc36\ud835\udc4fwill be compressed into one entry according to their compression weights and learnable\npositional biases \ud835\udc35\ud835\udc4e,\ud835\udc35\ud835\udc4f\u2208R\ud835\udc5a\u00d7\ud835\udc50, producing \ud835\udc36Comp\u2208R\ud835\udc5b\n\ud835\udc5a\u00d7\ud835\udc50. Each compressed entry \ud835\udc36Comp\n\ud835\udc56\u2208R\ud835\udc50is\ncomputed by\n[\ud835\udc46\ud835\udc4e\n\ud835\udc5a\ud835\udc56:\ud835\udc5a(\ud835\udc56+1)\u22121;\ud835\udc46\ud835\udc4f\n\ud835\udc5a(\ud835\udc56\u22121):\ud835\udc5a\ud835\udc56\u22121]=Softmax row([\ud835\udc4d\ud835\udc4e\n\ud835\udc5a\ud835\udc56:\ud835\udc5a(\ud835\udc56+1)\u22121+\ud835\udc35\ud835\udc4e;\ud835\udc4d\ud835\udc4f\n\ud835\udc5a(\ud835\udc56\u22121):\ud835\udc5a\ud835\udc56\u22121+\ud835\udc35\ud835\udc4f]), (11)\n\ud835\udc36Comp\n\ud835\udc56=\ud835\udc5a(\ud835\udc56+1)\u22121\u2211\ufe01\n\ud835\udc57=\ud835\udc5a\ud835\udc56\ud835\udc46\ud835\udc4e\n\ud835\udc57\u2299\ud835\udc36\ud835\udc4e\n\ud835\udc57+\ud835\udc5a\ud835\udc56\u22121\u2211\ufe01\n\ud835\udc57=\ud835\udc5a(\ud835\udc56\u22121)\ud835\udc46\ud835\udc4f\n\ud835\udc57\u2299\ud835\udc36\ud835\udc4f\n\ud835\udc57, (12)\nwhere\u2299denotes the Hadamard product; Softmax row(\u00b7)denotes the softmax operation along\nthe row dimension, which performs normalization across the total of 2 \ud835\udc5aelements from both\n\ud835\udc4d\ud835\udc4eand\ud835\udc4d\ud835\udc4f. When\ud835\udc56=0,\ud835\udc4d\ud835\udc4f\n\ud835\udc5a(\ud835\udc56\u22121):\ud835\udc5a\ud835\udc56\u22121is padded with negative infinity and \ud835\udc36\ud835\udc4f\n\ud835\udc5a(\ud835\udc56\u22121):\ud835\udc5a\ud835\udc56\u22121is padded\nwith zeros. Note that each \ud835\udc36Comp\n\ud835\udc56is derived from 2 \ud835\udc5aKV entries, but the indexes of \ud835\udc36\ud835\udc4fused for\n\ud835\udc36Comp\n\ud835\udc56and the indexes of\ud835\udc36\ud835\udc4eused for\ud835\udc36Comp\n\ud835\udc56\u22121are overlapped. Therefore, CSA in fact compresses\nthe sequence length to1\n\ud835\udc5atimes.\nLightning Indexer for Sparse Selection.After obtaining the compressed KV entries \ud835\udc36Comp,\nCSA applies the DSA strategy to select top-k compressed KV entries for core attention. First,\nCSA performs the same compression operation used for \ud835\udc36Compto get compressed indexer keys\n\ud835\udc3eIComp\u2208R\ud835\udc5b\n\ud835\udc5a\u00d7\ud835\udc50\ud835\udc3c, where\ud835\udc50\ud835\udc3cis the indexer head dimension. Then, for a query token \ud835\udc61, we produce\nthe indexer queries{q\ud835\udc3c\n\ud835\udc61,1;q\ud835\udc3c\n\ud835\udc61,2; ...;q\ud835\udc3c\n\ud835\udc61,\ud835\udc5b\ud835\udc3c\n\u210e}in a low-rank manner:\nc\ud835\udc44\n\ud835\udc61=h\ud835\udc61\u00b7\ud835\udc4a\ud835\udc37\ud835\udc44, (13)\n[q\ud835\udc3c\n\ud835\udc61,1;q\ud835\udc3c\n\ud835\udc61,2; ...;q\ud835\udc3c\n\ud835\udc61,\ud835\udc5b\ud835\udc3c\n\u210e]=q\ud835\udc3c\n\ud835\udc61=c\ud835\udc44\n\ud835\udc61\u00b7\ud835\udc4a\ud835\udc3c\ud835\udc48\ud835\udc44, (14)\nwhereh \ud835\udc61\u2208R\ud835\udc51is the input hidden state of the query token \ud835\udc61;c\ud835\udc44\n\ud835\udc61\u2208R\ud835\udc51\ud835\udc50is the compressed\nlatent vector for queries; \ud835\udc51\ud835\udc50denotes the query compression dimension; \ud835\udc5b\ud835\udc3c\n\u210edenotes the number\nof indexer query heads; \ud835\udc4a\ud835\udc37\ud835\udc44\u2208R\ud835\udc51\u00d7\ud835\udc51\ud835\udc50and\ud835\udc4a\ud835\udc3c\ud835\udc48\ud835\udc44\u2208R\ud835\udc51\ud835\udc50\u00d7\ud835\udc50\ud835\udc3c\ud835\udc5b\ud835\udc3c\n\u210eare the down-projection and up-\nprojection matrices for indexer queries, respectively. Next, the index score \ud835\udc3c\ud835\udc61,\ud835\udc60\u2208R between the\nquery token\ud835\udc61and a preceding compressed block\ud835\udc60(\ud835\udc60< Floor(\ud835\udc61\n\ud835\udc5a)) is computed by\n[\ud835\udc64\ud835\udc3c\n\ud835\udc61,1;\ud835\udc64\ud835\udc3c\n\ud835\udc61,2; ...;\ud835\udc64\ud835\udc3c\n\ud835\udc61,\ud835\udc5b\ud835\udc3c\n\u210e]=w\ud835\udc3c\n\ud835\udc61=h\ud835\udc61\u00b7\ud835\udc4a\ud835\udc64, (15)\n\ud835\udc3c\ud835\udc61,\ud835\udc60=\ud835\udc5b\ud835\udc3c\n\u210e\u2211\ufe01\n\u210e=1\ud835\udc64\ud835\udc3c\n\ud835\udc61,\u210e\u00b7ReLU\u0010\nq\ud835\udc3c\n\ud835\udc61,\u210e\u00b7\ud835\udc3eIComp\n\ud835\udc60\u0011\n, (16)\nwhere\ud835\udc4a\ud835\udc64\u2208R\ud835\udc51\u00d7\ud835\udc5b\ud835\udc3c\n\u210eis a learnable matrix; \ud835\udc64\ud835\udc3c\n\ud835\udc61,\u210e\u2208R is the weight of the \u210e-th indexer head. For a\nquery token \ud835\udc61, given its index scores \ud835\udc3c\ud835\udc61,:, we employ a top-k selector to selectively retain a subset\nof compressed KV entriesCSprsComp\n\ud835\udc61 for subsequent core attention:\nCSprsComp\n\ud835\udc61 =n\n\ud835\udc36Comp\n\ud835\udc60\f\f\f\ud835\udc3c\ud835\udc61,\ud835\udc60\u2208Top-k(\ud835\udc3c \ud835\udc61,:)o\n. (17)\n10\u2026\nHidden States of KV Tokens\u2026\nHidden State of Query TokenHeavily\nCompressed \nKV EntriesShared Key -Value Multi -Query Attention\nQueriesSliding Window\nKV EntriesConcatenation\nToken- Level\nCompressorFigure 4|Core architectures of HCA. It performs heavier compression, where the KV entries of\n\ud835\udc5a\u2032(\u226b\ud835\udc5a ) tokens will be consolidated into one. Also, we additionally introduce a small set of\nsliding window KV entries to enhance local fine-grained dependencies.\nShared Key-Value MQA.After selecting the sparse KV entries, CSA then performs core\nattention in a Multi-Query Attention (MQA) (Shazeer, 2019) manner, where each compressed\nKV entry inCSprsComp\n\ud835\udc61 serves as both attention key and value. To be specific, for a query token \ud835\udc61,\nwe first produce attention queries{q \ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e}from the compressed latent vectorc\ud835\udc44\n\ud835\udc61:\n[q\ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e]=q\ud835\udc61=c\ud835\udc44\n\ud835\udc61\u00b7\ud835\udc4a\ud835\udc48\ud835\udc44, (18)\nwhere\ud835\udc5b\u210edenotes the number of query heads; \ud835\udc4a\ud835\udc48\ud835\udc44\u2208R\ud835\udc51\ud835\udc50\u00d7\ud835\udc50\ud835\udc5b\u210eis the up-projection matrices for\nqueries. Note that the latent query vectorc\ud835\udc44\n\ud835\udc61is shared with that used for the indexer queries.\nNext, we perform MQA on{q \ud835\udc61,\ud835\udc56}andCSprsComp\n\ud835\udc61 :\no\ud835\udc61,\ud835\udc56=CoreAttn\u0010\nquery=q \ud835\udc61,\ud835\udc56,key=CSprsComp\n\ud835\udc61 ,value=CSprsComp\n\ud835\udc61\u0011\n, (19)\nwhereo \ud835\udc61,\ud835\udc56\u2208R\ud835\udc50is the core attention output of the \ud835\udc56-th head at the \ud835\udc61-th token; CoreAttn(\u00b7) denotes\nthe core attention operation.\nGrouped Output Projection.In the configuration of DeepSeek-V4, \ud835\udc50\ud835\udc5b\u210eis quite large. Therefore,\ndirectly projecting the outputs of the core attention operation [o\ud835\udc61,1;o\ud835\udc61,2; ...;o\ud835\udc61,\ud835\udc5b\u210e]=o\ud835\udc61\u2208R\ud835\udc50\ud835\udc5b\u210eto a\n\ud835\udc51-dimensional hidden state will impose a substantial computational burden. To mitigate this\ncost, we design a grouped output projection strategy. To be specific, we first split \ud835\udc5b\u210eoutputs\ninto\ud835\udc54groups, and then for each group of outputo\ud835\udc3a\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54, we project it to a \ud835\udc51\ud835\udc54-dimensional\nintermediate outputo\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc51\ud835\udc54, where\ud835\udc51\ud835\udc54< \ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54. Finally, we project the intermediate output\n[o\ud835\udc3a\u2032\n\ud835\udc61,1;o\ud835\udc3a\u2032\n\ud835\udc61,2; ...;o\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc54]\u2208R\ud835\udc51\ud835\udc54\ud835\udc54to the final attention output\u02c6 o \ud835\udc61\u2208R\ud835\udc51.\n2.3.2. Heavily Compressed Attention\nThe core architecture of HCA is illustrated in Figure 4, which compresses the KV cache in a\nheavier manner, but does not employ sparse attention.\nCompressed Key-Value Entries.By and large, the compression strategy of HCA is similar to\nthat of CSA, but employs a larger compression rate \ud835\udc5a\u2032(\u226b\ud835\udc5a ) and does not perform overlapped\n11" | |
| }, | |
| { | |
| "title": "Heavily Compressed Attention", | |
| "node_id": "0007", | |
| "start_index": 11, | |
| "end_index": 12, | |
| "summary": "The partial document discusses advanced attention mechanisms in neural networks, focusing on two core architectures: Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). It details how these methods compress key-value (KV) token entries to reduce computational complexity while maintaining effective attention. CSA uses a sparse selection of KV entries combined with Multi-Query Attention (MQA) and grouped output projection to efficiently handle large numbers of query heads. HCA further increases compression by consolidating many KV entries into fewer ones without sparse attention, also employing shared KV MQA and grouped output projection to manage computational costs. The document also highlights additional techniques such as sliding window KV entries for capturing local dependencies, normalization of queries and KV entries to stabilize training, and the use of learnable parameters for compression and projection. Overall, it presents a detailed explanation of how these compressed attention architectures optimize performance and efficiency in transformer models.", | |
| "text": "\u2026\nHidden States of KV Tokens\u2026\nHidden State of Query TokenHeavily\nCompressed \nKV EntriesShared Key -Value Multi -Query Attention\nQueriesSliding Window\nKV EntriesConcatenation\nToken- Level\nCompressorFigure 4|Core architectures of HCA. It performs heavier compression, where the KV entries of\n\ud835\udc5a\u2032(\u226b\ud835\udc5a ) tokens will be consolidated into one. Also, we additionally introduce a small set of\nsliding window KV entries to enhance local fine-grained dependencies.\nShared Key-Value MQA.After selecting the sparse KV entries, CSA then performs core\nattention in a Multi-Query Attention (MQA) (Shazeer, 2019) manner, where each compressed\nKV entry inCSprsComp\n\ud835\udc61 serves as both attention key and value. To be specific, for a query token \ud835\udc61,\nwe first produce attention queries{q \ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e}from the compressed latent vectorc\ud835\udc44\n\ud835\udc61:\n[q\ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e]=q\ud835\udc61=c\ud835\udc44\n\ud835\udc61\u00b7\ud835\udc4a\ud835\udc48\ud835\udc44, (18)\nwhere\ud835\udc5b\u210edenotes the number of query heads; \ud835\udc4a\ud835\udc48\ud835\udc44\u2208R\ud835\udc51\ud835\udc50\u00d7\ud835\udc50\ud835\udc5b\u210eis the up-projection matrices for\nqueries. Note that the latent query vectorc\ud835\udc44\n\ud835\udc61is shared with that used for the indexer queries.\nNext, we perform MQA on{q \ud835\udc61,\ud835\udc56}andCSprsComp\n\ud835\udc61 :\no\ud835\udc61,\ud835\udc56=CoreAttn\u0010\nquery=q \ud835\udc61,\ud835\udc56,key=CSprsComp\n\ud835\udc61 ,value=CSprsComp\n\ud835\udc61\u0011\n, (19)\nwhereo \ud835\udc61,\ud835\udc56\u2208R\ud835\udc50is the core attention output of the \ud835\udc56-th head at the \ud835\udc61-th token; CoreAttn(\u00b7) denotes\nthe core attention operation.\nGrouped Output Projection.In the configuration of DeepSeek-V4, \ud835\udc50\ud835\udc5b\u210eis quite large. Therefore,\ndirectly projecting the outputs of the core attention operation [o\ud835\udc61,1;o\ud835\udc61,2; ...;o\ud835\udc61,\ud835\udc5b\u210e]=o\ud835\udc61\u2208R\ud835\udc50\ud835\udc5b\u210eto a\n\ud835\udc51-dimensional hidden state will impose a substantial computational burden. To mitigate this\ncost, we design a grouped output projection strategy. To be specific, we first split \ud835\udc5b\u210eoutputs\ninto\ud835\udc54groups, and then for each group of outputo\ud835\udc3a\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54, we project it to a \ud835\udc51\ud835\udc54-dimensional\nintermediate outputo\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc51\ud835\udc54, where\ud835\udc51\ud835\udc54< \ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54. Finally, we project the intermediate output\n[o\ud835\udc3a\u2032\n\ud835\udc61,1;o\ud835\udc3a\u2032\n\ud835\udc61,2; ...;o\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc54]\u2208R\ud835\udc51\ud835\udc54\ud835\udc54to the final attention output\u02c6 o \ud835\udc61\u2208R\ud835\udc51.\n2.3.2. Heavily Compressed Attention\nThe core architecture of HCA is illustrated in Figure 4, which compresses the KV cache in a\nheavier manner, but does not employ sparse attention.\nCompressed Key-Value Entries.By and large, the compression strategy of HCA is similar to\nthat of CSA, but employs a larger compression rate \ud835\udc5a\u2032(\u226b\ud835\udc5a ) and does not perform overlapped\n11compression. Let \ud835\udc3b\u2208R\ud835\udc5b\u00d7\ud835\udc51be a sequence of input hidden states, HCA first computes the\noriginal KV entries\ud835\udc36\u2208R\ud835\udc5b\u00d7\ud835\udc50and their corresponding compression weights\ud835\udc4d\u2208R\ud835\udc5b\u00d7\ud835\udc50:\n\ud835\udc36=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc3e\ud835\udc49, (20)\n\ud835\udc4d=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc4d, (21)\nwhere\ud835\udc4a\ud835\udc3e\ud835\udc49,\ud835\udc4a\ud835\udc4d\u2208R\ud835\udc51\u00d7\ud835\udc50are trainable parameters. Next, each \ud835\udc5a\u2032KV entries in \ud835\udc36will be compressed\ninto one according to the compression weights and learnable positional biases \ud835\udc35\u2208R\ud835\udc5a\u2032\u00d7\ud835\udc50,\nproducing\ud835\udc36Comp\u2208R\ud835\udc5b\n\ud835\udc5a\u2032\u00d7\ud835\udc50. Each compressed entry\ud835\udc36Comp\n\ud835\udc56\u2208R\ud835\udc50is computed by\n\ud835\udc46\ud835\udc5a\u2032\ud835\udc56:\ud835\udc5a\u2032(\ud835\udc56+1)\u22121 =Softmax row(\ud835\udc4d\ud835\udc5a\u2032\ud835\udc56:\ud835\udc5a\u2032(\ud835\udc56+1)\u22121+\ud835\udc35), (22)\n\ud835\udc36Comp\n\ud835\udc56=\ud835\udc5a\u2032(\ud835\udc56+1)\u22121\u2211\ufe01\n\ud835\udc57=\ud835\udc5a\u2032\ud835\udc56\ud835\udc46\ud835\udc57\u2299\ud835\udc36\ud835\udc57. (23)\nThrough this compression operation, HCA compresses the sequence length to1\n\ud835\udc5a\u2032times.\nShared Key-Value MQA and Grouped Output Projection.HCA also employs the shared KV\nMQA and grouped output projection strategies as CSA does. After the KV compression, for a\nquery token\ud835\udc61, HCA first produces attention queries{q \ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e}in a low-rank manner:\nc\ud835\udc44\n\ud835\udc61=h\ud835\udc61\u00b7\ud835\udc4a\ud835\udc37\ud835\udc44, (24)\n[q\ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e]=q\ud835\udc61=c\ud835\udc44\n\ud835\udc61\u00b7\ud835\udc4a\ud835\udc48\ud835\udc44, (25)\nwhereh \ud835\udc61\u2208R\ud835\udc51is the input hidden state of the query token \ud835\udc61;\ud835\udc5b\u210edenotes the number of query\nheads;\ud835\udc4a\ud835\udc37\ud835\udc44\u2208R\ud835\udc51\u00d7\ud835\udc51\ud835\udc50and\ud835\udc4a\ud835\udc48\ud835\udc44\u2208R\ud835\udc51\ud835\udc50\u00d7\ud835\udc50\ud835\udc5b\u210eare the down-projection and up-projection matrices for\nqueries, respectively. Next, we perform MQA on{q \ud835\udc61,\ud835\udc56}and\ud835\udc36Comp:\no\ud835\udc61,\ud835\udc56=CoreAttn\u0010\nquery=q \ud835\udc61,\ud835\udc56,key=\ud835\udc36Comp,value=\ud835\udc36Comp\u0011\n, (26)\nwhereo \ud835\udc61,\ud835\udc56\u2208R\ud835\udc50is the core attention output of the \ud835\udc56-th head at the \ud835\udc61-th token. Next, as CSA does,\nHCA splits \ud835\udc5b\u210eoutputs into \ud835\udc54groups, and for each group of outputo\ud835\udc3a\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54, HCA projects it\nto a\ud835\udc51\ud835\udc54-dimensional intermediate outputo\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc51\ud835\udc54, where\ud835\udc51\ud835\udc54< \ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54. Finally, HCA projects the\nintermediate output[o\ud835\udc3a\u2032\n\ud835\udc61,1;o\ud835\udc3a\u2032\n\ud835\udc61,2; ...;o\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc54]\u2208R\ud835\udc51\ud835\udc54\ud835\udc54to the final attention output\u02c6 o \ud835\udc61\u2208R\ud835\udc51.\n2.3.3. Other Details\nIn addition to the core architectures of CSA and HCA described above, our hybrid attention\nincorporates several other techniques. For writing clarity, we omit these additional techniques\nfrom the above introduction and will briefly describe them in this subsection. Also, this subsec-\ntion focuses only on the core ideas of them and may omit some tiny details for simplicity. We\nencourage the readers to refer to our open-source implementation for unambiguous details.\nQuery and Key-Value Entry Normalization.For both CSA and HCA, we perform an addi-\ntional RMSNorm operation on each head of the queries and the only head of the compressed KV\nentries, just before the core attention operation. This normalization avoids exploding attention\nlogits and may improve training stability.\n12" | |
| }, | |
| { | |
| "title": "Other Details", | |
| "node_id": "0008", | |
| "start_index": 12, | |
| "end_index": 13, | |
| "summary": "The partial document primarily discusses the architecture and techniques behind a hybrid attention mechanism combining Compressed Sequence Attention (CSA) and Hybrid Compressed Attention (HCA) for efficient long-context language modeling. It details the process of compressing key-value (KV) entries using trainable parameters and positional biases to reduce sequence length, followed by multi-query attention (MQA) with shared KV and grouped output projection strategies. Additional components include normalization of queries and KV entries to stabilize training, partial application of Rotary Positional Embedding (RoPE) to incorporate relative positional information, and a supplementary sliding window attention branch to capture local dependencies while preserving causality. The document also introduces an attention sink mechanism allowing flexible adjustment of attention scores. Finally, it highlights efficiency improvements in the DeepSeek-V4 series, such as mixed-precision storage formats (BF16 and FP8), low-precision attention computation (FP4), reduced attention top-k, and the significant reduction of KV cache size and computational FLOPs compared to previous models, enabling scalable and efficient attention over extremely long contexts.", | |
| "text": "compression. Let \ud835\udc3b\u2208R\ud835\udc5b\u00d7\ud835\udc51be a sequence of input hidden states, HCA first computes the\noriginal KV entries\ud835\udc36\u2208R\ud835\udc5b\u00d7\ud835\udc50and their corresponding compression weights\ud835\udc4d\u2208R\ud835\udc5b\u00d7\ud835\udc50:\n\ud835\udc36=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc3e\ud835\udc49, (20)\n\ud835\udc4d=\ud835\udc3b\u00b7\ud835\udc4a\ud835\udc4d, (21)\nwhere\ud835\udc4a\ud835\udc3e\ud835\udc49,\ud835\udc4a\ud835\udc4d\u2208R\ud835\udc51\u00d7\ud835\udc50are trainable parameters. Next, each \ud835\udc5a\u2032KV entries in \ud835\udc36will be compressed\ninto one according to the compression weights and learnable positional biases \ud835\udc35\u2208R\ud835\udc5a\u2032\u00d7\ud835\udc50,\nproducing\ud835\udc36Comp\u2208R\ud835\udc5b\n\ud835\udc5a\u2032\u00d7\ud835\udc50. Each compressed entry\ud835\udc36Comp\n\ud835\udc56\u2208R\ud835\udc50is computed by\n\ud835\udc46\ud835\udc5a\u2032\ud835\udc56:\ud835\udc5a\u2032(\ud835\udc56+1)\u22121 =Softmax row(\ud835\udc4d\ud835\udc5a\u2032\ud835\udc56:\ud835\udc5a\u2032(\ud835\udc56+1)\u22121+\ud835\udc35), (22)\n\ud835\udc36Comp\n\ud835\udc56=\ud835\udc5a\u2032(\ud835\udc56+1)\u22121\u2211\ufe01\n\ud835\udc57=\ud835\udc5a\u2032\ud835\udc56\ud835\udc46\ud835\udc57\u2299\ud835\udc36\ud835\udc57. (23)\nThrough this compression operation, HCA compresses the sequence length to1\n\ud835\udc5a\u2032times.\nShared Key-Value MQA and Grouped Output Projection.HCA also employs the shared KV\nMQA and grouped output projection strategies as CSA does. After the KV compression, for a\nquery token\ud835\udc61, HCA first produces attention queries{q \ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e}in a low-rank manner:\nc\ud835\udc44\n\ud835\udc61=h\ud835\udc61\u00b7\ud835\udc4a\ud835\udc37\ud835\udc44, (24)\n[q\ud835\udc61,1;q\ud835\udc61,2; ...;q\ud835\udc61,\ud835\udc5b\u210e]=q\ud835\udc61=c\ud835\udc44\n\ud835\udc61\u00b7\ud835\udc4a\ud835\udc48\ud835\udc44, (25)\nwhereh \ud835\udc61\u2208R\ud835\udc51is the input hidden state of the query token \ud835\udc61;\ud835\udc5b\u210edenotes the number of query\nheads;\ud835\udc4a\ud835\udc37\ud835\udc44\u2208R\ud835\udc51\u00d7\ud835\udc51\ud835\udc50and\ud835\udc4a\ud835\udc48\ud835\udc44\u2208R\ud835\udc51\ud835\udc50\u00d7\ud835\udc50\ud835\udc5b\u210eare the down-projection and up-projection matrices for\nqueries, respectively. Next, we perform MQA on{q \ud835\udc61,\ud835\udc56}and\ud835\udc36Comp:\no\ud835\udc61,\ud835\udc56=CoreAttn\u0010\nquery=q \ud835\udc61,\ud835\udc56,key=\ud835\udc36Comp,value=\ud835\udc36Comp\u0011\n, (26)\nwhereo \ud835\udc61,\ud835\udc56\u2208R\ud835\udc50is the core attention output of the \ud835\udc56-th head at the \ud835\udc61-th token. Next, as CSA does,\nHCA splits \ud835\udc5b\u210eoutputs into \ud835\udc54groups, and for each group of outputo\ud835\udc3a\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54, HCA projects it\nto a\ud835\udc51\ud835\udc54-dimensional intermediate outputo\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc56\u2208R\ud835\udc51\ud835\udc54, where\ud835\udc51\ud835\udc54< \ud835\udc50\ud835\udc5b\u210e\n\ud835\udc54. Finally, HCA projects the\nintermediate output[o\ud835\udc3a\u2032\n\ud835\udc61,1;o\ud835\udc3a\u2032\n\ud835\udc61,2; ...;o\ud835\udc3a\u2032\n\ud835\udc61,\ud835\udc54]\u2208R\ud835\udc51\ud835\udc54\ud835\udc54to the final attention output\u02c6 o \ud835\udc61\u2208R\ud835\udc51.\n2.3.3. Other Details\nIn addition to the core architectures of CSA and HCA described above, our hybrid attention\nincorporates several other techniques. For writing clarity, we omit these additional techniques\nfrom the above introduction and will briefly describe them in this subsection. Also, this subsec-\ntion focuses only on the core ideas of them and may omit some tiny details for simplicity. We\nencourage the readers to refer to our open-source implementation for unambiguous details.\nQuery and Key-Value Entry Normalization.For both CSA and HCA, we perform an addi-\ntional RMSNorm operation on each head of the queries and the only head of the compressed KV\nentries, just before the core attention operation. This normalization avoids exploding attention\nlogits and may improve training stability.\n12Partial Rotary Positional Embedding.For both CSA and HCA, we partially employ the Rotary\nPositional Embedding (RoPE) (Su et al., 2024) to the attention queries, KV entries, and the core\nattention outputs. To be specific, for each query vector and KV entry vector used in CSA and\nHCA, we apply RoPE to its last 64 dimensions. Since the KV entries serve as both attention\nkeys and values, the naive core attention outputs {o\ud835\udc61,\ud835\udc56}will carry absolute position embeddings,\nderived from the weighted sum of KV entries. As a countermeasure, we also apply RoPE with\nposition\u2212\ud835\udc56on the last 64 dimensions of eacho \ud835\udc61,\ud835\udc56. In this way, the output of the core attention\nwill also carry relative position embeddings \u2014 the contribution of each KV entry to the core\nattention outputs will also be related to the distance between the query and the KV entry.\nAdditional Branch of Sliding Window Attention.In order to strictly preserve causality in\nCSA and HCA, each query attends to only preceding compressed KV blocks. Consequently, a\nquery cannot access information from other tokens within its own compressed block. Meanwhile,\nrecent tokens usually possess greater relevance to the query token in language modeling. For\nthese reasons, we introduce a supplementary attention branch to both CSA and HCA in a sliding\nwindow manner, for better modeling of local dependencies. To be specific, for each query token,\nwe additionally produce \ud835\udc5bwinuncompressed KV entries corresponding to the recent \ud835\udc5bwintokens.\nIn the core attention of CSA and HCA, these KV entries in the sliding window will be used\nalong with the compressed KV entries.\nAttention Sink.In the core attention of CSA and HCA, we employ the trick of attention\nsink (OpenAI, 2025; Xiao et al., 2024). To be specific, we set a series of learnable sink logits\n{\ud835\udc67\u2032\n1,\ud835\udc67\u2032\n2, ...,\ud835\udc67\u2032\n\ud835\udc5b\u210e}. For the \u210e-th attention head, Exp(\ud835\udc67\u2032\n\u210e)will be added to the denominator of the\nattention score:\n\ud835\udc60\u210e,\ud835\udc56,\ud835\udc57=Exp(\ud835\udc67\u210e,\ud835\udc56,\ud835\udc57)\u00cd\n\ud835\udc58Exp(\ud835\udc67\u210e,\ud835\udc56,\ud835\udc58)+Exp(\ud835\udc67\u2032\n\u210e), (27)\nwhere\ud835\udc60\u210e,\ud835\udc56,\ud835\udc57,\ud835\udc67\u210e,\ud835\udc56,\ud835\udc57\u2208R denote the attention score and attention logit of the \u210e-th attention head\nbetween the \ud835\udc56-th query token and the \ud835\udc57-th preceding token or compressed block. This technique\nallows each query head to adjust its total attention scores to be not equal to 1, and even to be\nnear 0.\n2.3.4. Efficiency Discussion\nDue to the employment of hybrid CSA and HCA, together with low-precision computation\nand storage, the attention module of DeepSeek-V4 series achieves remarkable efficiency in both\nattention FLOPs and KV cache size, especially in long-context scenarios. First, we adopt a\nmixed storage format for KV entries: BF16 precision is used for the rotary positional embedding\n(RoPE) dimensions, while FP8 precision is applied to the remaining dimensions. This hybrid\nrepresentation reduces the KV cache size by nearly half compared with pure BF16 storage.\nSecond, attention computation within the lightning indexer is performed in FP4 precision,\nwhich accelerates the attention operation under extremely long contexts. Third, relative to\nDeepSeek-V3.2, a smaller attention top-k is chosen in DeepSeek-V4 series, thereby improving\nmodel efficiency on short- and medium-length texts. Finally, and most importantly, compressed\nattention and hybrid attention techniques substantially reduce both the KV cache size and the\ncomputational FLOPs.\nTaking BF16 GQA8 (Ainslie et al., 2023) with a head dimension of 128 as the baseline \u2014 one\nof the common configurations of LLM attention \u2014 the KV cache size of DeepSeek-V4 series can\nbe dramatically reduced to approximately 2% times of that baseline in the 1M-context setting.\n13" | |
| }, | |
| { | |
| "title": "Efficiency Discussion", | |
| "node_id": "0009", | |
| "start_index": 13, | |
| "end_index": 14, | |
| "summary": "The partial document primarily discusses advanced techniques and optimizations used in the DeepSeek-V4 series attention mechanism and training process. Key points include:\n\n1. **Partial Rotary Positional Embedding (RoPE):** The document explains the application of RoPE to the last 64 dimensions of attention queries, key-value (KV) entries, and core attention outputs in both Compressed Sliding Attention (CSA) and Hybrid Compressed Attention (HCA). This approach ensures that the attention outputs carry relative positional information, addressing the issue of absolute position embeddings in naive attention outputs.\n\n2. **Additional Sliding Window Attention Branch:** To maintain strict causality while improving local dependency modeling, a supplementary sliding window attention branch is introduced. This allows each query token to attend to recent uncompressed KV entries within a sliding window, complementing the compressed KV blocks.\n\n3. **Attention Sink Mechanism:** The document describes the use of learnable sink logits added to the denominator of attention scores, enabling each attention head to adjust total attention weights flexibly, including allowing scores to approach zero, enhancing attention dynamics.\n\n4. **Efficiency Enhancements:** Several strategies are outlined to improve computational and memory efficiency, including:\n - A hybrid storage format for KV entries using BF16 precision for RoPE dimensions and FP8 for others, reducing cache size by nearly half.\n - FP4 precision computation in the lightning indexer for faster attention operations in long contexts.\n - Reduced attention top-k size compared to previous versions for better efficiency on shorter texts.\n - Overall, compressed and hybrid attention techniques significantly reduce KV cache size and FLOPs, achieving dramatic efficiency gains (e.g., KV cache size reduced to about 2% of a common baseline in 1M-context scenarios).\n\n5. **Muon Optimizer:** The document introduces the Muon optimizer used for most DeepSeek-V4 modules, highlighting its faster convergence and improved training stability. It details:\n - The retention of AdamW for specific modules while applying Muon elsewhere.\n - Use of weight decay, the Nesterov momentum trick, and RMS rescaling for update stability.\n - A novel hybrid Newton-Schulz iteration method for approximate orthogonalization of weight update matrices, involving two stages with different coefficient sets to ensure rapid convergence and precise stabilization of singular values.\n - The architecture\u2019s ability to apply RMSNorm directly on attention queries and KV entries to prevent exploding attention logits, eliminating the need for QK-Clip techniques.\n\nOverall, the document focuses on sophisticated positional embedding strategies, attention mechanism improvements, efficiency optimizations, and a novel optimizer design to enhance the performance and scalability of the DeepSeek-V4 model series.", | |
| "text": "Partial Rotary Positional Embedding.For both CSA and HCA, we partially employ the Rotary\nPositional Embedding (RoPE) (Su et al., 2024) to the attention queries, KV entries, and the core\nattention outputs. To be specific, for each query vector and KV entry vector used in CSA and\nHCA, we apply RoPE to its last 64 dimensions. Since the KV entries serve as both attention\nkeys and values, the naive core attention outputs {o\ud835\udc61,\ud835\udc56}will carry absolute position embeddings,\nderived from the weighted sum of KV entries. As a countermeasure, we also apply RoPE with\nposition\u2212\ud835\udc56on the last 64 dimensions of eacho \ud835\udc61,\ud835\udc56. In this way, the output of the core attention\nwill also carry relative position embeddings \u2014 the contribution of each KV entry to the core\nattention outputs will also be related to the distance between the query and the KV entry.\nAdditional Branch of Sliding Window Attention.In order to strictly preserve causality in\nCSA and HCA, each query attends to only preceding compressed KV blocks. Consequently, a\nquery cannot access information from other tokens within its own compressed block. Meanwhile,\nrecent tokens usually possess greater relevance to the query token in language modeling. For\nthese reasons, we introduce a supplementary attention branch to both CSA and HCA in a sliding\nwindow manner, for better modeling of local dependencies. To be specific, for each query token,\nwe additionally produce \ud835\udc5bwinuncompressed KV entries corresponding to the recent \ud835\udc5bwintokens.\nIn the core attention of CSA and HCA, these KV entries in the sliding window will be used\nalong with the compressed KV entries.\nAttention Sink.In the core attention of CSA and HCA, we employ the trick of attention\nsink (OpenAI, 2025; Xiao et al., 2024). To be specific, we set a series of learnable sink logits\n{\ud835\udc67\u2032\n1,\ud835\udc67\u2032\n2, ...,\ud835\udc67\u2032\n\ud835\udc5b\u210e}. For the \u210e-th attention head, Exp(\ud835\udc67\u2032\n\u210e)will be added to the denominator of the\nattention score:\n\ud835\udc60\u210e,\ud835\udc56,\ud835\udc57=Exp(\ud835\udc67\u210e,\ud835\udc56,\ud835\udc57)\u00cd\n\ud835\udc58Exp(\ud835\udc67\u210e,\ud835\udc56,\ud835\udc58)+Exp(\ud835\udc67\u2032\n\u210e), (27)\nwhere\ud835\udc60\u210e,\ud835\udc56,\ud835\udc57,\ud835\udc67\u210e,\ud835\udc56,\ud835\udc57\u2208R denote the attention score and attention logit of the \u210e-th attention head\nbetween the \ud835\udc56-th query token and the \ud835\udc57-th preceding token or compressed block. This technique\nallows each query head to adjust its total attention scores to be not equal to 1, and even to be\nnear 0.\n2.3.4. Efficiency Discussion\nDue to the employment of hybrid CSA and HCA, together with low-precision computation\nand storage, the attention module of DeepSeek-V4 series achieves remarkable efficiency in both\nattention FLOPs and KV cache size, especially in long-context scenarios. First, we adopt a\nmixed storage format for KV entries: BF16 precision is used for the rotary positional embedding\n(RoPE) dimensions, while FP8 precision is applied to the remaining dimensions. This hybrid\nrepresentation reduces the KV cache size by nearly half compared with pure BF16 storage.\nSecond, attention computation within the lightning indexer is performed in FP4 precision,\nwhich accelerates the attention operation under extremely long contexts. Third, relative to\nDeepSeek-V3.2, a smaller attention top-k is chosen in DeepSeek-V4 series, thereby improving\nmodel efficiency on short- and medium-length texts. Finally, and most importantly, compressed\nattention and hybrid attention techniques substantially reduce both the KV cache size and the\ncomputational FLOPs.\nTaking BF16 GQA8 (Ainslie et al., 2023) with a head dimension of 128 as the baseline \u2014 one\nof the common configurations of LLM attention \u2014 the KV cache size of DeepSeek-V4 series can\nbe dramatically reduced to approximately 2% times of that baseline in the 1M-context setting.\n13Algorithm 1Muon Optimizer for DeepSeek-V4\nRequire:Learning rate\ud835\udf02, momentum\ud835\udf07, weight decay\ud835\udf06, update rescaling factor\ud835\udefe\n1:foreach training step\ud835\udc61do\n2:foreach logically independent weight\ud835\udc4a\u2208R\ud835\udc5b\u00d7\ud835\udc5ado\n3:\ud835\udc3a \ud835\udc61=\u2207\ud835\udc4aL\ud835\udc61(\ud835\udc4a\ud835\udc61\u22121)\u22b2Compute gradients\n4:\ud835\udc40 \ud835\udc61=\ud835\udf07\ud835\udc40\ud835\udc61\u22121+\ud835\udc3a\ud835\udc61 \u22b2Accumulate momentum buffer\n5:\ud835\udc42\u2032\n\ud835\udc61=HybridNewtonSchulz(\ud835\udf07\ud835\udc40 \ud835\udc61+\ud835\udc3a\ud835\udc61)\u22b2Nesterov trick and hybrid Newton-Schulz\n6:\ud835\udc42 \ud835\udc61=\ud835\udc42\u2032\n\ud835\udc61\u00b7\u221a\ufe01\nmax(\ud835\udc5b,\ud835\udc5a)\u00b7\ud835\udefe \u22b2Rescale the update RMS\n7:\ud835\udc4a \ud835\udc61=\ud835\udc4a\ud835\udc61\u22121\u00b7(1\u2212\ud835\udf02\ud835\udf06)\u2212\ud835\udf02\ud835\udc42\ud835\udc61 \u22b2Perform weight decay and update\n8:end for\n9:end for\nMoreover, even when compared with DeepSeek-V3.2 (DeepSeek-AI, 2025) \u2014 already an efficient\nbaseline \u2014 DeepSeek-V4 series still exhibits substantial advantages in efficiency. A comparison\nof their inference FLOPs and KV cache size is provided in the right part of Figure 1.\n2.4. Muon Optimizer\nWe employ the Muon (Jordan et al., 2024; Liu et al., 2025) optimizer for the majority of modules\nin DeepSeek-V4 series due to its faster convergence and improved training stability. The full\nalgorithm of our Muon optimization is summarized in Algorithm 1.\nBasic Configurations.We maintain the AdamW (Loshchilov and Hutter, 2017) optimizer for\nthe embedding module, the prediction head module, the static biases and gating factors of\nmHC modules, and the weights of all RMSNorm modules. All other modules are updated with\nMuon. Following Liu et al. (2025), we also apply weight decay to Muon parameters, use the\nNesterov (Jordan et al., 2024; Nesterov, 1983) trick, and rescale the Root Mean Square (RMS) of\nthe update matrix for reutilization of our AdamW hyper-parameters. Different from them, we\nuse hybrid Newton-Schulz iterations for orthogonalization.\nHybrid Newton-Schulz Iterations.For a given matrix \ud835\udc40, let its Singular Value Decomposition\n(SVD) be\ud835\udc40=\ud835\udc48\u03a3\ud835\udc49\ud835\udc47. The Newton-Schulz iterations aim to approximately orthogonalize \ud835\udc40to be\n\ud835\udc48\ud835\udc49\ud835\udc47. Usually, \ud835\udc40will be first normalized as \ud835\udc400=\ud835\udc40/||\ud835\udc40|| \ud835\udc39to ensure its maximum singular value\ndoes not exceed 1. Then, each Newton-Schulz iteration performs the following operation:\n\ud835\udc40\ud835\udc58=\ud835\udc4e\ud835\udc40\ud835\udc58\u22121+\ud835\udc4f(\ud835\udc40\ud835\udc58\u22121\ud835\udc40\ud835\udc47\n\ud835\udc58\u22121)\ud835\udc40\ud835\udc58\u22121+\ud835\udc50(\ud835\udc40\ud835\udc58\u22121\ud835\udc40\ud835\udc47\n\ud835\udc58\u22121)2\ud835\udc40\ud835\udc58\u22121. (28)\nOur hybrid Newton-Schulz performs 10 iterations over two distinct stages. During the first 8\nsteps, we use coefficients (\ud835\udc4e,\ud835\udc4f,\ud835\udc50)=( 3.4445,\u22124.7750, 2.0315)to drive rapid convergence, bringing\nthe singular values close to 1. In the final 2 steps, we switch to coefficients (\ud835\udc4e,\ud835\udc4f,\ud835\udc50)=( 2,\u22121.5, 0.5),\nwhich stabilize the singular values precisely at 1.\nAvoiding Exploding Attention Logits.The attention architecture of DeepSeek-V4 series al-\nlows us to directly apply RMSNorm on the attention queries and KV entries, which effectively\nprevents attention logits from exploding. Consequently, we do not employ the QK-Clip tech-\nnique (Liu et al., 2025) in our Muon optimizer.\n14" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Muon Optimizer", | |
| "node_id": "0010", | |
| "start_index": 14, | |
| "end_index": 14, | |
| "summary": "The partial document primarily covers the Muon optimizer used in the DeepSeek-V4 series, detailing its algorithmic steps and advantages over previous versions like DeepSeek-V3.2. It explains the integration of the Muon optimizer with specific modules, while retaining AdamW for others, and highlights enhancements such as the use of weight decay, the Nesterov momentum trick, and RMS update rescaling. A significant focus is on the hybrid Newton-Schulz iterations employed for orthogonalizing update matrices, describing the two-stage iterative process with specific coefficients for rapid convergence and stabilization. Additionally, the document discusses how the DeepSeek-V4 attention architecture leverages RMSNorm to prevent exploding attention logits, eliminating the need for QK-Clip techniques. Overall, it emphasizes improved training stability, faster convergence, and efficiency gains in DeepSeek-V4 through these optimization strategies.", | |
| "text": "Algorithm 1Muon Optimizer for DeepSeek-V4\nRequire:Learning rate\ud835\udf02, momentum\ud835\udf07, weight decay\ud835\udf06, update rescaling factor\ud835\udefe\n1:foreach training step\ud835\udc61do\n2:foreach logically independent weight\ud835\udc4a\u2208R\ud835\udc5b\u00d7\ud835\udc5ado\n3:\ud835\udc3a \ud835\udc61=\u2207\ud835\udc4aL\ud835\udc61(\ud835\udc4a\ud835\udc61\u22121)\u22b2Compute gradients\n4:\ud835\udc40 \ud835\udc61=\ud835\udf07\ud835\udc40\ud835\udc61\u22121+\ud835\udc3a\ud835\udc61 \u22b2Accumulate momentum buffer\n5:\ud835\udc42\u2032\n\ud835\udc61=HybridNewtonSchulz(\ud835\udf07\ud835\udc40 \ud835\udc61+\ud835\udc3a\ud835\udc61)\u22b2Nesterov trick and hybrid Newton-Schulz\n6:\ud835\udc42 \ud835\udc61=\ud835\udc42\u2032\n\ud835\udc61\u00b7\u221a\ufe01\nmax(\ud835\udc5b,\ud835\udc5a)\u00b7\ud835\udefe \u22b2Rescale the update RMS\n7:\ud835\udc4a \ud835\udc61=\ud835\udc4a\ud835\udc61\u22121\u00b7(1\u2212\ud835\udf02\ud835\udf06)\u2212\ud835\udf02\ud835\udc42\ud835\udc61 \u22b2Perform weight decay and update\n8:end for\n9:end for\nMoreover, even when compared with DeepSeek-V3.2 (DeepSeek-AI, 2025) \u2014 already an efficient\nbaseline \u2014 DeepSeek-V4 series still exhibits substantial advantages in efficiency. A comparison\nof their inference FLOPs and KV cache size is provided in the right part of Figure 1.\n2.4. Muon Optimizer\nWe employ the Muon (Jordan et al., 2024; Liu et al., 2025) optimizer for the majority of modules\nin DeepSeek-V4 series due to its faster convergence and improved training stability. The full\nalgorithm of our Muon optimization is summarized in Algorithm 1.\nBasic Configurations.We maintain the AdamW (Loshchilov and Hutter, 2017) optimizer for\nthe embedding module, the prediction head module, the static biases and gating factors of\nmHC modules, and the weights of all RMSNorm modules. All other modules are updated with\nMuon. Following Liu et al. (2025), we also apply weight decay to Muon parameters, use the\nNesterov (Jordan et al., 2024; Nesterov, 1983) trick, and rescale the Root Mean Square (RMS) of\nthe update matrix for reutilization of our AdamW hyper-parameters. Different from them, we\nuse hybrid Newton-Schulz iterations for orthogonalization.\nHybrid Newton-Schulz Iterations.For a given matrix \ud835\udc40, let its Singular Value Decomposition\n(SVD) be\ud835\udc40=\ud835\udc48\u03a3\ud835\udc49\ud835\udc47. The Newton-Schulz iterations aim to approximately orthogonalize \ud835\udc40to be\n\ud835\udc48\ud835\udc49\ud835\udc47. Usually, \ud835\udc40will be first normalized as \ud835\udc400=\ud835\udc40/||\ud835\udc40|| \ud835\udc39to ensure its maximum singular value\ndoes not exceed 1. Then, each Newton-Schulz iteration performs the following operation:\n\ud835\udc40\ud835\udc58=\ud835\udc4e\ud835\udc40\ud835\udc58\u22121+\ud835\udc4f(\ud835\udc40\ud835\udc58\u22121\ud835\udc40\ud835\udc47\n\ud835\udc58\u22121)\ud835\udc40\ud835\udc58\u22121+\ud835\udc50(\ud835\udc40\ud835\udc58\u22121\ud835\udc40\ud835\udc47\n\ud835\udc58\u22121)2\ud835\udc40\ud835\udc58\u22121. (28)\nOur hybrid Newton-Schulz performs 10 iterations over two distinct stages. During the first 8\nsteps, we use coefficients (\ud835\udc4e,\ud835\udc4f,\ud835\udc50)=( 3.4445,\u22124.7750, 2.0315)to drive rapid convergence, bringing\nthe singular values close to 1. In the final 2 steps, we switch to coefficients (\ud835\udc4e,\ud835\udc4f,\ud835\udc50)=( 2,\u22121.5, 0.5),\nwhich stabilize the singular values precisely at 1.\nAvoiding Exploding Attention Logits.The attention architecture of DeepSeek-V4 series al-\nlows us to directly apply RMSNorm on the attention queries and KV entries, which effectively\nprevents attention logits from exploding. Consequently, we do not employ the QK-Clip tech-\nnique (Liu et al., 2025) in our Muon optimizer.\n14" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "General Infrastructures", | |
| "node_id": "0011", | |
| "start_index": 15, | |
| "end_index": 14, | |
| "summary": "The partial document discusses the main points covered, focusing on summarizing key topics or themes presented within the given text segment.", | |
| "text": "", | |
| "nodes": [ | |
| { | |
| "title": "Fine-Grained Communication-Computation Overlap in Expert Parallelism", | |
| "node_id": "0012", | |
| "start_index": 15, | |
| "end_index": 16, | |
| "summary": "The partial document focuses on advanced techniques to optimize Mixture-of-Experts (MoE) model parallelism, specifically through Expert Parallelism (EP). It introduces a fine-grained communication-computation overlap scheme that fuses these two processes into a single pipelined kernel, effectively hiding communication latency beneath computation. The approach involves splitting experts into smaller waves to enable concurrent communication and computation, improving throughput and reducing interconnect bandwidth demands. Performance evaluations demonstrate significant speedups on NVIDIA GPUs and Huawei Ascend NPUs, with an open-sourced CUDA mega-kernel implementation provided. The document also shares insights and hardware design recommendations, emphasizing the importance of balancing computation and communication volumes, power budget considerations, communication primitives, and activation function choices to maximize efficiency. Additionally, it briefly mentions the use of TileLang for developing fused kernels to manage the complexity of numerous fine-grained operators in the model architecture.", | |
| "text": "3. General Infrastructures\n3.1. Fine-Grained Communication-Computation Overlap in Expert Parallelism\nMixture-of-Experts (MoE) can be accelerated via Expert Parallelism (EP). However, EP re-\nquires complex inter-node communication and imposes substantial demands on interconnect\nbandwidth and latency. To alleviate the communication bottleneck in EP and achieve higher\nend-to-end performance under lower interconnection bandwidth requirements, we propose\na fine-grained EP scheme that fuses communication and computation into a single pipelined\nkernel for communication-computation overlapping.\nCommunication Latency Can Be Hidden.The key insight of our EP scheme is that the\ncommunication latency can be effectively hidden beneath computation in MoE layers. As shown\nin Figure 5, in DeepSeek-V4 series, each MoE layer can be decomposed mainly into four stages:\ntwo communication-bound stages, Dispatch and Combine , and two computation-bound stages,\nLinear-1 and Linear-2 . Our profiling reveals that within a single MoE layer, the total time of\ncommunication is less than that of the computation. Therefore, after fusing communication and\ncomputation into a unified pipeline, computation remains the dominant bottleneck, implying\nthat the system can tolerate lower interconnect bandwidth without degrading end-to-end\nperformance.\nL1 Act L2(a) Naive Solution\nCommunication\nComputation L1 Act L2Theoretical speedup: 1.42\u00d7(b) Comet\nDispatch\nComputation\nActivation\n& CombineL1 L2 L1 L2 L1 L2\nAct Act Act\nExpert Wave 1 Expert Wave 2 Expert Wave 3Theoretical speedup: 1.92\u00d7(c) Ours\nDispatch All-to-All\nLinear 1 GEMM\nSwiGLU + FP8 Cast\nLinear 2 GEMM\nCombine All-to-All\nFigure 5|Illustration of our EP scheme with related works. Comet (Zhang et al., 2025b) overlaps\nDispatch with Linear-1, and Linear-2 with Combine, separately. Our EP scheme achieves a finer-\ngrained overlapping by splitting and scheduling experts into waves. The theoretical speedup is\nevaluated in the configuration of the DeepSeek-V4-Flash architecture.\nFine-Grained EP Scheme.To further lower the interconnect bandwidth requirement and\namplify the benefits of overlapping, we introduce a finer-grained expert partitioning scheme.\nInspired by many related works (Aimuyo et al., 2025; Zhang et al., 2025b), we split and schedule\nthe experts into waves . Each wave consists of a small portion of experts. As soon as all experts\nwithin the wave have completed their communication, computation can commence immediately\nwithout waiting for other experts. In steady state, computation of current wave, token transfer for\nthe next wave, and result sending of completed experts all proceed concurrently, as demonstrated\nin Figure 5. This forms a fine-grained pipeline among experts, keeping both computation and\ncommunication continuous throughout the wave. The wave-based scheduling speeds up the\n15performance on extreme cases such as Reinforcement Learning (RL) rollout, which usually\nencounters long-tail small batches.\nPerformance and Open-Sourced Mega-Kernel.We validated the fine-grained EP scheme\non both NVIDIA GPUs and HUAWEI Ascend NPUs platforms. Compared against strong\nnon-fused baselines, it achieves 1.50 \u223c1.73\u00d7speedup for general inference workloads, and\nup to 1.96\u00d7for latency-sensitive scenarios such as RL rollouts and high-speed agent serving.\nWe have open-sourced the CUDA-based mega-kernel implementation namedMegaMoE2as a\ncomponent of DeepGEMM.\nObservations and Proposals.We share observations and lessons from kernel development\nand offer some proposals to hardware vendors, in the hope of aiding efficient hardware design\nand achieving better software-hardware co-design:\n\u2022Computation-Communication Ratio.Full communication-computation overlap hinges\non the computation-communication ratio, rather than the bandwidth solely. Denoting peak\ncompute throughput as \ud835\udc36and interconnect bandwidth as \ud835\udc35, communication can be fully\nhidden when \ud835\udc36/\ud835\udc35\u2a7d\ud835\udc49 comp/\ud835\udc49comm , where\ud835\udc49comp denotes the computation volume and \ud835\udc49comm\nrefers to the communication volume. For DeepSeek-V4-Pro, where each token-expert\npair requires 6 \u210e\ud835\udc51FLOPs (SwiGLU gate, up, and down projections) but only 3 \u210ebytes of\ncommunication (FP8 Dispatch + BF16 Combine), this simplifies to:\n\ud835\udc36\n\ud835\udc35\u2a7d2\ud835\udc51=6144 FLOPs/Byte.\nThat is, each GBps of interconnect bandwidth suffices to hide the communication for 6.1\nTFLOP/s of compute. Once bandwidth meets this threshold, it ceases to be the bottleneck,\nand devoting additional silicon area to further bandwidth brings diminishing returns.\nWe encourage future hardware designs to target such balance points rather than scale\nbandwidth unconditionally.\n\u2022Power Budget.Extreme kernel fusion drives compute, memory, and network to high\nload simultaneously, making power throttling a key performance limiter. We suggest that\nfuture hardware designs provide sufficient power headroom for such fully concurrent\nworkloads.\n\u2022Communication Primitives.In the dispatch stage, we adopt a pull-based approach where\neach GPU actively reads activations from remote GPUs, avoiding the high notification\nlatency that fine-grained push entails. Future hardware with lower-latency cross-GPU\nsignaling would make push viable and enable more natural communication patterns.\n\u2022Activation Function.We propose replacing SwiGLU with a low-cost element-wise ac-\ntivation that involves no exponential or division operations. This directly reduces the\noverhead of post-GEMM processing, preventing the GEMM pipeline from being stalled\nby activation function computation, thereby enhancing overall computational throughput\nand resource utilization.\n3.2. Flexible and Efficient Kernel Development with TileLang\nIn practice, our elaborate model architecture would have resulted in hundreds of fine-grained\nTorch ATen operators. We adopt TileLang (Wang et al., 2026) to develop a set of fused kernels\n2https://github.com/deepseek-ai/DeepGEMM/pull/304\n16" | |
| }, | |
| { | |
| "title": "Flexible and Efficient Kernel Development with TileLang", | |
| "node_id": "0013", | |
| "start_index": 16, | |
| "end_index": 18, | |
| "summary": "The partial document discusses advanced techniques and observations related to optimizing performance for machine learning workloads, particularly focusing on reinforcement learning scenarios with long-tail small batches. It presents the development and validation of a fine-grained kernel fusion scheme, achieving significant speedups on NVIDIA GPUs and HUAWEI Ascend NPUs, and introduces an open-sourced CUDA-based mega-kernel called MegaMoE2 as part of DeepGEMM. Key hardware-software co-design insights are shared, emphasizing the importance of balancing computation-communication ratios, providing sufficient power budgets for concurrent workloads, improving communication primitives, and proposing more efficient activation functions to enhance throughput.\n\nThe document also details the use of TileLang, a domain-specific language for flexible and efficient kernel development, which enables rapid prototyping, deep optimizations, and reduced CPU-side invocation overhead through host code generation. It highlights the integration of the Z3 SMT solver for formal integer analysis to improve compiler optimizations and discusses design choices prioritizing numerical precision and bitwise reproducibility without sacrificing performance.\n\nFurthermore, the document covers the creation of high-performance, batch-invariant, and deterministic kernel libraries to ensure reproducibility and consistency across training and inference. It addresses challenges in achieving batch invariance in attention mechanisms and matrix multiplication, proposing novel dual-kernel strategies and replacing traditional libraries with DeepGEMM to maintain bitwise identical outputs regardless of batch position, thereby supporting debugging, stability, and consistent post-training behavior.", | |
| "text": "performance on extreme cases such as Reinforcement Learning (RL) rollout, which usually\nencounters long-tail small batches.\nPerformance and Open-Sourced Mega-Kernel.We validated the fine-grained EP scheme\non both NVIDIA GPUs and HUAWEI Ascend NPUs platforms. Compared against strong\nnon-fused baselines, it achieves 1.50 \u223c1.73\u00d7speedup for general inference workloads, and\nup to 1.96\u00d7for latency-sensitive scenarios such as RL rollouts and high-speed agent serving.\nWe have open-sourced the CUDA-based mega-kernel implementation namedMegaMoE2as a\ncomponent of DeepGEMM.\nObservations and Proposals.We share observations and lessons from kernel development\nand offer some proposals to hardware vendors, in the hope of aiding efficient hardware design\nand achieving better software-hardware co-design:\n\u2022Computation-Communication Ratio.Full communication-computation overlap hinges\non the computation-communication ratio, rather than the bandwidth solely. Denoting peak\ncompute throughput as \ud835\udc36and interconnect bandwidth as \ud835\udc35, communication can be fully\nhidden when \ud835\udc36/\ud835\udc35\u2a7d\ud835\udc49 comp/\ud835\udc49comm , where\ud835\udc49comp denotes the computation volume and \ud835\udc49comm\nrefers to the communication volume. For DeepSeek-V4-Pro, where each token-expert\npair requires 6 \u210e\ud835\udc51FLOPs (SwiGLU gate, up, and down projections) but only 3 \u210ebytes of\ncommunication (FP8 Dispatch + BF16 Combine), this simplifies to:\n\ud835\udc36\n\ud835\udc35\u2a7d2\ud835\udc51=6144 FLOPs/Byte.\nThat is, each GBps of interconnect bandwidth suffices to hide the communication for 6.1\nTFLOP/s of compute. Once bandwidth meets this threshold, it ceases to be the bottleneck,\nand devoting additional silicon area to further bandwidth brings diminishing returns.\nWe encourage future hardware designs to target such balance points rather than scale\nbandwidth unconditionally.\n\u2022Power Budget.Extreme kernel fusion drives compute, memory, and network to high\nload simultaneously, making power throttling a key performance limiter. We suggest that\nfuture hardware designs provide sufficient power headroom for such fully concurrent\nworkloads.\n\u2022Communication Primitives.In the dispatch stage, we adopt a pull-based approach where\neach GPU actively reads activations from remote GPUs, avoiding the high notification\nlatency that fine-grained push entails. Future hardware with lower-latency cross-GPU\nsignaling would make push viable and enable more natural communication patterns.\n\u2022Activation Function.We propose replacing SwiGLU with a low-cost element-wise ac-\ntivation that involves no exponential or division operations. This directly reduces the\noverhead of post-GEMM processing, preventing the GEMM pipeline from being stalled\nby activation function computation, thereby enhancing overall computational throughput\nand resource utilization.\n3.2. Flexible and Efficient Kernel Development with TileLang\nIn practice, our elaborate model architecture would have resulted in hundreds of fine-grained\nTorch ATen operators. We adopt TileLang (Wang et al., 2026) to develop a set of fused kernels\n2https://github.com/deepseek-ai/DeepGEMM/pull/304\n16to replace the vast majority of them, delivering optimal performance with minimal effort. It\nalso allows us to quickly prototype operators like attention variants during validation. These\nkernels play critical roles in model architecture development, large -scale training, and ultimately\nproduction deployment of inference services. As a Domain-Specific Language (DSL), TileLang\nbalances development productivity with runtime efficiency, enabling rapid development while\nsupporting deep, iterative optimizations within the same codebase. Additionally, we collab-\norate closely with the TileLang community to foster a more agile, efficient, and stable kernel\ndevelopment workflow.\nReducing Invocation Overhead with Host Codegen.As accelerators continue to grow in\nperformance, CPU-side orchestration overhead becomes increasingly prominent. For small,\nhighly optimized kernels, such fixed host overhead can easily cap utilization and throughput.\nA common source of this overhead is that host-side logic, such as runtime contract checks, is\ntypically written in Python for flexibility and thus incurs a fixed per-invocation cost.\nWe mitigate this overhead withHost Codegen, which moves most host-side logic into gen-\nerated host code. Specifically, we first co-generate the device kernel and a lightweight host\nlauncher at the IR (Intermediate Representation) level, embedding the necessary metadata\u2014such\nas data types, rank/shape constraints, and stride/layout assumptions\u2014parsed from the lan-\nguage frontend. The launcher is then lowered to the host source code built on top of the\nTVM-FFI (Chen et al., 2018) framework, whose compact calling convention and zero-copy tensor\ninterop together minimize host-side overhead. At runtime, this generated host code performs\nvalidation and argument marshaling, shifting all per -invocation checks out of the Python exe-\ncution path. Our measurements show that CPU-side validation overhead drops from tens or\nhundreds of microseconds to less than one microsecond per invocation.\nSMT-Solver-Assisted Formal Integer Analysis.TileLang kernels involve complex tensor\nindex arithmetic that requires strong formal integer analysis. During compilation passes such\nas layout inference, memory hazard detection, and bound analysis, the compiler must verify\nwhether integer expressions satisfy specific properties to enable the corresponding optimiza-\ntions. Therefore, stronger formal analysis capabilities can unlock more advanced and complex\noptimization opportunities.\nTo this end, we integrate the Z3 SMT solver (De Moura and Bj\u00f8rner, 2008) into TileLang\u2019s\nalgebraic system, providing formal analysis capability for most integer expressions in tensor\nprograms. We strike a balance between computational overhead and formal expressiveness by\ntranslating TileLang\u2019s integer expressions into Z3\u2019s quantifier-free non-linear integer arithmetic\n(QF_NIA). Based on Integer Linear Programming (ILP) solvers, QF_NIA seamlessly resolves\nstandard linear integer expressions common in kernels. Furthermore, its inherent non-linear\nreasoning capacity effectively addresses advanced challenges like vectorization over variable\ntensor shapes. Under reasonable resource limits, Z3 elevates overall optimization performance\nwhile restricting compilation time overhead to just a few seconds. The impact is substantial\nacross multiple passes, including vectorization, barrier insertion, and code simplification.\nNumerical Precision and Bitwise Reproducibility.In production settings, numerical correct-\nness and reproducibility are as critical as raw throughput. We therefore prioritize accuracy by\ndefault: fast -math optimizations are disabled at the compiler level, and precision -affecting ap-\nproximations are provided only as explicit, opt-in frontend operators (e.g., T.__exp ,T.__log ,\nandT.__sin ). Conversely, when strict IEEE-754 semantics are required, TileLang provides\n17IEEE-compliant intrinsics with explicit rounding modes (e.g., T.ieee_fsqrt ,T.ieee_fdiv ,\nandT.ieee_add), enabling developers to precisely specify numerical behavior.\nWe also target bitwise reproducibility for validating kernels against hand -written CUDA\nbaselines. We align TileLang\u2019s algebraic simplification and lowering rules with mainstream\nCUDA toolchains (e.g., NVCC) to avoid transformations that introduce unintended bit -level\ndifferences. Layout annotations (e.g., T.annotate_layout ) further allow users to pin down\nlayout-dependent lowering decisions, keeping evaluation and accumulation order consistent\nwith the reference CUDA implementation and thus enabling bit-identical outputs when desired.\nOur evaluation shows that these accuracy- and reproducibility-oriented design choices do\nnot sacrifice performance: under conservative defaults, TileLang kernels remain competitive,\nwhile exposing knobs to selectively relax numerical constraints for higher speed.\n3.3. High-Performance Batch-Invariant and Deterministic Kernel Libraries\nTo enable efficient training and inference, we develop a comprehensive set of high-performance\ncomputational kernels. Beyond basic functionalities and maximizing hardware utilization,\nanother pivotal design goal is to ensure training reproducibility and bitwise alignment among\npre-training, post-training, and inference pipelines. Therefore, we implement end-to-end,\nbitwise batch-invariant, and deterministic kernels with minimal performance overhead. These\nkernels are helpful for debugging, stability analysis, and consistent post-training behavior.\nBatch Invariance.Batch invariance ensures that the output of any given token remains bitwise\nidentical, regardless of its position within a batch. To implement batch invariance, the primary\nchallenges are listed as follows:\n\u2022Attention.To achieve batch invariance, we cannot use the split-KV method (Dao et al.,\n2023), which distributes the attention computation for a single sequence across multiple\nStream Multiprocessors (SMs) to balance the load of SMs. However, abandoning this\ntechnique will lead to severe wave-quantization problems3, which can adversely affect\nGPU utilization. To address this, we develop a dual-kernel strategy for batch-invariant\ndecoding. The first kernel computes the attention output for an entire sequence within\na single SM, ensuring high throughput for fully occupied waves. The second kernel, to\nminimize the latency of the final partially-filled wave and thus alleviate wave-quantization,\nuses multiple SMs for a single sequence. For the bitwise identity of these two kernels,\nwe carefully design the calculation path of the second kernel to ensure its accumulation\norder is the same as that of the first kernel. Additionally, the second kernel utilizes dis-\ntributed shared memory4within thread-block clusters, enabling high-speed data exchange\nacross SMs. This dual-kernel method effectively confines the overhead of batch-invariant\ndecoding to be negligible.\n\u2022Matrix Multiplication.Traditional cuBLAS library (NVIDIA Corporation, 2024) cannot\nachieve batch invariance. Therefore, we replace it end-to-end with DeepGEMM (Zhao\net al., 2025). Furthermore, for very small batch sizes, conventional implementation usually\nemploys split-k (Osama et al., 2023) techniques to improve performance. Unfortunately,\nsplit-k techniques cannot guarantee batch invariance, a pivotal feature in DeepSeek-V4.\n3https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplicat\nion/index.html#wave-quant\n4https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels\n.html#distributed-shared-memory\n18" | |
| }, | |
| { | |
| "title": "High-Performance Batch-Invariant and Deterministic Kernel Libraries", | |
| "node_id": "0014", | |
| "start_index": 18, | |
| "end_index": 19, | |
| "summary": "The partial document discusses the design and implementation of a high-performance, reproducible computational framework for training and inference in deep learning models. Key points include:\n\n1. **IEEE-compliant Intrinsics and Bitwise Reproducibility:** The framework uses IEEE-compliant intrinsics with explicit rounding modes to ensure precise numerical behavior. It aligns algebraic simplifications and lowering rules with mainstream CUDA toolchains to avoid unintended bit-level differences, enabling bit-identical outputs when desired. Layout annotations help maintain consistent evaluation and accumulation order.\n\n2. **Batch-Invariant and Deterministic Kernel Libraries:** The document details the development of computational kernels that are both batch-invariant and deterministic, ensuring bitwise identical outputs regardless of token position in a batch. This includes:\n - A dual-kernel strategy for batch-invariant attention computation that balances GPU utilization and maintains bitwise identity.\n - Replacement of traditional cuBLAS matrix multiplication with DeepGEMM to achieve batch invariance, avoiding split-k techniques that break this property, while introducing optimizations to maintain performance.\n - Techniques to ensure determinism in training by addressing non-deterministic accumulation orders caused by atomic operations, particularly in attention backward passes, Mixture of Experts (MoE) backward passes, and small-batch matrix multiplications.\n\n3. **Training Framework Enhancements:** Building on a scalable infrastructure from a previous version, the framework incorporates innovations to support new architectural components such as the Muon optimizer, mHC, and hybrid attention mechanisms, while maintaining training efficiency and stability.\n\n4. **Efficient Muon Optimizer Implementation:** The document describes a hybrid ZeRO bucket assignment strategy tailored for the Muon optimizer, which requires full gradient matrices. This approach balances memory load across ranks using a knapsack algorithm and padding to optimize reduce-scatter operations, incurring minimal memory overhead.", | |
| "text": "IEEE-compliant intrinsics with explicit rounding modes (e.g., T.ieee_fsqrt ,T.ieee_fdiv ,\nandT.ieee_add), enabling developers to precisely specify numerical behavior.\nWe also target bitwise reproducibility for validating kernels against hand -written CUDA\nbaselines. We align TileLang\u2019s algebraic simplification and lowering rules with mainstream\nCUDA toolchains (e.g., NVCC) to avoid transformations that introduce unintended bit -level\ndifferences. Layout annotations (e.g., T.annotate_layout ) further allow users to pin down\nlayout-dependent lowering decisions, keeping evaluation and accumulation order consistent\nwith the reference CUDA implementation and thus enabling bit-identical outputs when desired.\nOur evaluation shows that these accuracy- and reproducibility-oriented design choices do\nnot sacrifice performance: under conservative defaults, TileLang kernels remain competitive,\nwhile exposing knobs to selectively relax numerical constraints for higher speed.\n3.3. High-Performance Batch-Invariant and Deterministic Kernel Libraries\nTo enable efficient training and inference, we develop a comprehensive set of high-performance\ncomputational kernels. Beyond basic functionalities and maximizing hardware utilization,\nanother pivotal design goal is to ensure training reproducibility and bitwise alignment among\npre-training, post-training, and inference pipelines. Therefore, we implement end-to-end,\nbitwise batch-invariant, and deterministic kernels with minimal performance overhead. These\nkernels are helpful for debugging, stability analysis, and consistent post-training behavior.\nBatch Invariance.Batch invariance ensures that the output of any given token remains bitwise\nidentical, regardless of its position within a batch. To implement batch invariance, the primary\nchallenges are listed as follows:\n\u2022Attention.To achieve batch invariance, we cannot use the split-KV method (Dao et al.,\n2023), which distributes the attention computation for a single sequence across multiple\nStream Multiprocessors (SMs) to balance the load of SMs. However, abandoning this\ntechnique will lead to severe wave-quantization problems3, which can adversely affect\nGPU utilization. To address this, we develop a dual-kernel strategy for batch-invariant\ndecoding. The first kernel computes the attention output for an entire sequence within\na single SM, ensuring high throughput for fully occupied waves. The second kernel, to\nminimize the latency of the final partially-filled wave and thus alleviate wave-quantization,\nuses multiple SMs for a single sequence. For the bitwise identity of these two kernels,\nwe carefully design the calculation path of the second kernel to ensure its accumulation\norder is the same as that of the first kernel. Additionally, the second kernel utilizes dis-\ntributed shared memory4within thread-block clusters, enabling high-speed data exchange\nacross SMs. This dual-kernel method effectively confines the overhead of batch-invariant\ndecoding to be negligible.\n\u2022Matrix Multiplication.Traditional cuBLAS library (NVIDIA Corporation, 2024) cannot\nachieve batch invariance. Therefore, we replace it end-to-end with DeepGEMM (Zhao\net al., 2025). Furthermore, for very small batch sizes, conventional implementation usually\nemploys split-k (Osama et al., 2023) techniques to improve performance. Unfortunately,\nsplit-k techniques cannot guarantee batch invariance, a pivotal feature in DeepSeek-V4.\n3https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplicat\nion/index.html#wave-quant\n4https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels\n.html#distributed-shared-memory\n18Therefore, we abandon split-k in most scenarios, which, however, may cause performance\ndegradation. To address this, we introduce a set of optimizations that enable our imple-\nmentation of matrix multiplication to match or even surpass the performance of standard\nsplit-k in most major scenarios.\nDeterminism.Deterministic training is highly beneficial for debugging hardware or software\nissues. Moreover, when training exhibits anomalies such as loss spikes, determinism enables\nresearchers to more easily pinpoint numerical causes and further refine the model design. Non-\ndeterminism in training typically stems from non-deterministic accumulation order, often due\nto the use of atomic addition instructions. This issue primarily occurs during the backward pass,\nnotably at the following parts:\n\u2022Attention Backward.In conventional implementations of backward propagation for\nsparse attention, we use atomicAdd to accumulate gradients for the KV tokens. This\nintroduces non-determinism due to the non-associativity of floating-point addition. To\naddress this problem, we allocate separate accumulation buffers for each SM, followed by\na global deterministic summation across all buffers.\n\u2022MoE Backward.When multiple SMs from different ranks concurrently write data to\nthe same buffer on a receiving rank, negotiating writing positions also introduces non-\ndeterminism. To resolve this, we design a token order pre-processing mechanism within\neach single rank, combined with buffer isolation across multiple ranks. This strategy\nensures determinism of both the send results of expert parallelism and the accumulation\norder in the MoE backward pass.\n\u2022Matrix Multiplication inmHC.mHC involves a matrix multiplication with an output di-\nmension of only 24. For very small batch sizes, we are compelled to use the split-k (Osama\net al., 2023) algorithm, whose naive implementation will cause non-determinism. To\novercome this, we output each split part separately and perform a deterministic reduction\nin a subsequent kernel, thereby preserving both performance and determinism.\n3.4. Training Framework\nOur training framework is built upon the scalable and efficient infrastructure developed for\nDeepSeek-V3 (DeepSeek-AI, 2024). In training DeepSeek-V4, we inherit this robust foundation\nwhile introducing several key innovations to accommodate its novel architectural components \u2014\nspecifically the Muon optimizer,mHC, and the hybrid attention mechanism \u2014 while maintaining\nhigh training efficiency and stability.\n3.4.1. Efficient Implementation of Muon\nThe Muon optimizer requires the full gradient matrix to compute parameter updates, which\npresents a challenge when combined with the Zero Redundancy Optimizer (ZeRO) (Rajbhandari\net al., 2020). Traditional ZeRO is designed for element-wise optimizers like AdamW, where a\nsingle parameter matrix can be partitioned and updated across multiple ranks. To address this\nconflict, we design a hybrid strategy of ZeRO bucket assignment for Muon.\nFor dense parameters, we limit the maximum size of ZeRO parallelism and employ a\nknapsack algorithm to assign parameter matrices to these ranks, ensuring each rank manages a\nroughly balanced load. The bucket on each rank is padded to match the size of the largest bucket\nacross ranks, facilitating efficient reduce-scatter operations. This padding typically incurs less\nthan 10% memory overhead in our setup, where each rank manages no more than five parameter\n19" | |
| }, | |
| { | |
| "title": "Training Framework", | |
| "node_id": "0015", | |
| "start_index": 19, | |
| "end_index": 19, | |
| "summary": "The partial document discusses optimizations and strategies to improve the performance and determinism of matrix multiplication and training processes in advanced machine learning models. It highlights the abandonment of the split-k algorithm due to its drawbacks and introduces alternative optimizations that achieve equal or better performance. The document emphasizes the importance of deterministic training for debugging and model refinement, detailing specific solutions to eliminate non-determinism in backward passes of sparse attention, Mixture of Experts (MoE), and matrix multiplication in mHC by using techniques such as separate accumulation buffers, token order preprocessing, and deterministic reduction kernels. Additionally, it outlines the training framework built on DeepSeek-V3 infrastructure, adapted for DeepSeek-V4 with innovations like the Muon optimizer, mHC, and hybrid attention mechanisms. The text also addresses challenges in integrating the Muon optimizer with the Zero Redundancy Optimizer (ZeRO), proposing a hybrid ZeRO bucket assignment strategy using a knapsack algorithm to balance load and maintain efficiency with minimal memory overhead.", | |
| "text": "Therefore, we abandon split-k in most scenarios, which, however, may cause performance\ndegradation. To address this, we introduce a set of optimizations that enable our imple-\nmentation of matrix multiplication to match or even surpass the performance of standard\nsplit-k in most major scenarios.\nDeterminism.Deterministic training is highly beneficial for debugging hardware or software\nissues. Moreover, when training exhibits anomalies such as loss spikes, determinism enables\nresearchers to more easily pinpoint numerical causes and further refine the model design. Non-\ndeterminism in training typically stems from non-deterministic accumulation order, often due\nto the use of atomic addition instructions. This issue primarily occurs during the backward pass,\nnotably at the following parts:\n\u2022Attention Backward.In conventional implementations of backward propagation for\nsparse attention, we use atomicAdd to accumulate gradients for the KV tokens. This\nintroduces non-determinism due to the non-associativity of floating-point addition. To\naddress this problem, we allocate separate accumulation buffers for each SM, followed by\na global deterministic summation across all buffers.\n\u2022MoE Backward.When multiple SMs from different ranks concurrently write data to\nthe same buffer on a receiving rank, negotiating writing positions also introduces non-\ndeterminism. To resolve this, we design a token order pre-processing mechanism within\neach single rank, combined with buffer isolation across multiple ranks. This strategy\nensures determinism of both the send results of expert parallelism and the accumulation\norder in the MoE backward pass.\n\u2022Matrix Multiplication inmHC.mHC involves a matrix multiplication with an output di-\nmension of only 24. For very small batch sizes, we are compelled to use the split-k (Osama\net al., 2023) algorithm, whose naive implementation will cause non-determinism. To\novercome this, we output each split part separately and perform a deterministic reduction\nin a subsequent kernel, thereby preserving both performance and determinism.\n3.4. Training Framework\nOur training framework is built upon the scalable and efficient infrastructure developed for\nDeepSeek-V3 (DeepSeek-AI, 2024). In training DeepSeek-V4, we inherit this robust foundation\nwhile introducing several key innovations to accommodate its novel architectural components \u2014\nspecifically the Muon optimizer,mHC, and the hybrid attention mechanism \u2014 while maintaining\nhigh training efficiency and stability.\n3.4.1. Efficient Implementation of Muon\nThe Muon optimizer requires the full gradient matrix to compute parameter updates, which\npresents a challenge when combined with the Zero Redundancy Optimizer (ZeRO) (Rajbhandari\net al., 2020). Traditional ZeRO is designed for element-wise optimizers like AdamW, where a\nsingle parameter matrix can be partitioned and updated across multiple ranks. To address this\nconflict, we design a hybrid strategy of ZeRO bucket assignment for Muon.\nFor dense parameters, we limit the maximum size of ZeRO parallelism and employ a\nknapsack algorithm to assign parameter matrices to these ranks, ensuring each rank manages a\nroughly balanced load. The bucket on each rank is padded to match the size of the largest bucket\nacross ranks, facilitating efficient reduce-scatter operations. This padding typically incurs less\nthan 10% memory overhead in our setup, where each rank manages no more than five parameter\n19", | |
| "nodes": [ | |
| { | |
| "title": "Efficient Implementation of Muon", | |
| "node_id": "0016", | |
| "start_index": 19, | |
| "end_index": 20, | |
| "summary": "The partial document discusses several key technical optimizations and design strategies aimed at improving the performance, determinism, and efficiency of training large-scale neural network models. It begins by addressing the abandonment of the split-k algorithm due to its potential performance degradation and introduces alternative optimizations that match or surpass split-k performance in matrix multiplication. The document emphasizes the importance of deterministic training for debugging and model refinement, detailing solutions to non-determinism issues in backward passes of sparse attention, Mixture of Experts (MoE) models, and matrix multiplication in mHC by using buffer isolation, token order preprocessing, and deterministic reduction techniques.\n\nThe training framework builds upon a scalable infrastructure from a previous model version, incorporating innovations to support new components such as the Muon optimizer, mHC, and hybrid attention mechanisms while maintaining efficiency and stability. For the Muon optimizer, the document outlines a hybrid ZeRO parallelism strategy that balances memory and computation, including specialized handling of dense and MoE parameters, parameter merging for efficient Newton-Schulz iterations, and gradient quantization with a two-phase communication approach to preserve numerical robustness.\n\nRegarding mHC, the document describes cost-effective and memory-efficient implementations through fused kernels, selective recomputation of intermediate tensors, and adjustments to pipeline overlapping schemes, limiting overhead to a minimal fraction of training time. Finally, it addresses challenges in long-context attention with compressed attention mechanisms by proposing a two-stage communication protocol to handle token compression across partition boundaries, ensuring proper data exchange and compression across ranks.", | |
| "text": "Therefore, we abandon split-k in most scenarios, which, however, may cause performance\ndegradation. To address this, we introduce a set of optimizations that enable our imple-\nmentation of matrix multiplication to match or even surpass the performance of standard\nsplit-k in most major scenarios.\nDeterminism.Deterministic training is highly beneficial for debugging hardware or software\nissues. Moreover, when training exhibits anomalies such as loss spikes, determinism enables\nresearchers to more easily pinpoint numerical causes and further refine the model design. Non-\ndeterminism in training typically stems from non-deterministic accumulation order, often due\nto the use of atomic addition instructions. This issue primarily occurs during the backward pass,\nnotably at the following parts:\n\u2022Attention Backward.In conventional implementations of backward propagation for\nsparse attention, we use atomicAdd to accumulate gradients for the KV tokens. This\nintroduces non-determinism due to the non-associativity of floating-point addition. To\naddress this problem, we allocate separate accumulation buffers for each SM, followed by\na global deterministic summation across all buffers.\n\u2022MoE Backward.When multiple SMs from different ranks concurrently write data to\nthe same buffer on a receiving rank, negotiating writing positions also introduces non-\ndeterminism. To resolve this, we design a token order pre-processing mechanism within\neach single rank, combined with buffer isolation across multiple ranks. This strategy\nensures determinism of both the send results of expert parallelism and the accumulation\norder in the MoE backward pass.\n\u2022Matrix Multiplication inmHC.mHC involves a matrix multiplication with an output di-\nmension of only 24. For very small batch sizes, we are compelled to use the split-k (Osama\net al., 2023) algorithm, whose naive implementation will cause non-determinism. To\novercome this, we output each split part separately and perform a deterministic reduction\nin a subsequent kernel, thereby preserving both performance and determinism.\n3.4. Training Framework\nOur training framework is built upon the scalable and efficient infrastructure developed for\nDeepSeek-V3 (DeepSeek-AI, 2024). In training DeepSeek-V4, we inherit this robust foundation\nwhile introducing several key innovations to accommodate its novel architectural components \u2014\nspecifically the Muon optimizer,mHC, and the hybrid attention mechanism \u2014 while maintaining\nhigh training efficiency and stability.\n3.4.1. Efficient Implementation of Muon\nThe Muon optimizer requires the full gradient matrix to compute parameter updates, which\npresents a challenge when combined with the Zero Redundancy Optimizer (ZeRO) (Rajbhandari\net al., 2020). Traditional ZeRO is designed for element-wise optimizers like AdamW, where a\nsingle parameter matrix can be partitioned and updated across multiple ranks. To address this\nconflict, we design a hybrid strategy of ZeRO bucket assignment for Muon.\nFor dense parameters, we limit the maximum size of ZeRO parallelism and employ a\nknapsack algorithm to assign parameter matrices to these ranks, ensuring each rank manages a\nroughly balanced load. The bucket on each rank is padded to match the size of the largest bucket\nacross ranks, facilitating efficient reduce-scatter operations. This padding typically incurs less\nthan 10% memory overhead in our setup, where each rank manages no more than five parameter\n19matrices. When the overall size of data parallelism exceeds the limit for ZeRO, we compute\nthe Muon update redundantly across the extra data-parallel groups, trading computation for\nreduced total bucket memory.\nFor MoE parameters, we optimize each expert independently. We first flatten all down\nprojection matrices in SwiGLU (Shazeer, 2020) of all experts across all layers, followed by\nflattened up projection matrices and gate matrices. Then, we pad the flattened vector to ensure\nwe can evenly distribute this vector across all ranks without splitting any logically independent\nmatrix. Given the large number of experts, we do not impose a limit of ZeRO parallelism for\nMoE parameters, and the padding overhead is also negligible.\nAdditionally, on each rank, consecutive parameters of identical shape will be automatically\nmerged, enabling batched execution of the Newton-Schulz iterations for better hardware utiliza-\ntion. Furthermore, we observe that the Newton-Schulz iterations in Muon remain stable when\ncomputed with BF16 matrix multiplications. Leveraging this, we further quantize, in a stochastic\nrounding manner, the MoE gradients to be synchronized across data-parallel ranks to the BF16\nprecision, halving the communication volume. To avoid accumulation errors introduced by\nlow-precision adders, we replace conventional tree- or ring-based reduce-scatter collectives with\na two-phase approach. First, an all-to-all operation exchanges local gradients across ranks, and\nthen each rank performs a local sum in FP32. This design maintains numerical robustness.\n3.4.2. Cost-Effective and Memory-Efficient Implementation of mHC\nThe introduction ofmHC increases both activation memory consumption and communication\nvolume between pipeline stages, compared with conventional residual connections. To mitigate\nthese costs, we implement several optimization strategies.\nFirstly, we carefully design and implement fused kernels ofmHC for both training and\ninference. Secondly, we introduce a recomputation strategy that selectively checkpoints interme-\ndiate tensors. Specifically, we recompute most hidden states between layers and all normalized\nlayer inputs, while avoiding recomputation of compute-intensive operations. This achieves a\nbalance between memory saving and computational overhead. Thirdly, we adjust the DualPipe\n1F1B overlapping scheme to accommodate the increased pipeline communication and enable\nconcurrent execution of some operations inmHC.\nCollectively, these optimizations constrain the wall-time overhead ofmHC to only 6.7% of\nthe overlapped 1F1B pipeline stage. More details of the engineering optimization can be found\nin the dedicatedmHC paper (Xie et al., 2026).\n3.4.3. Contextual Parallelism for Long-Context Attention\nConventional Context Parallelism (CP) partitions the sequence dimension, with each rank\nmaintaining contiguous \ud835\udc60tokens. This introduces two challenges to our compressed attention\nmechanisms (i.e., CSA and HCA). On the one hand, training samples are packed from multiple\nsequences, and each sequence is compressed independently by a factor of \ud835\udc5a(or\ud835\udc5a\u2032), with any\ntrailing tokens fewer than \ud835\udc5abeing discarded. Consequently, the compressed KV lengths are\ntypically less than\ud835\udc60\n\ud835\udc5aand vary across ranks. On the other hand, the compression requires \ud835\udc5a\nconsecutive KV entries, which may straddle the boundary between two neighboring CP ranks.\nTo address these challenges, we design a two-stage communication approach. In the first\nstage, each rank \ud835\udc56sends its last \ud835\udc5auncompressed KV entries to rank \ud835\udc56+1. Then, rank \ud835\udc56+1\ncompresses some of these received entries together with its local \ud835\udc60uncompressed KV entries,\n20" | |
| }, | |
| { | |
| "title": "Cost-Effective and Memory-Efficient Implementation ofmHC", | |
| "node_id": "0017", | |
| "start_index": 20, | |
| "end_index": 20, | |
| "summary": "The partial document discusses advanced optimization techniques for large-scale model training, focusing on memory efficiency, parallelism, and communication strategies. It covers the handling of Muon updates in data parallelism by redundantly computing across extra groups to reduce memory usage. For Mixture of Experts (MoE) parameters, it details independent expert optimization through flattening and padding projection matrices to enable even distribution without splitting, avoiding ZeRO parallelism limits and minimizing padding overhead. The document also highlights merging parameters for batched Newton-Schulz iterations, leveraging BF16 precision for stable, efficient gradient synchronization with a novel two-phase reduce-scatter approach to maintain numerical accuracy. Additionally, it presents cost-effective implementations of mHC, including fused kernels, selective recomputation of tensors, and adjustments to pipeline overlapping schemes to reduce memory and communication overhead with minimal runtime impact. Finally, it addresses challenges in Contextual Parallelism for long-context attention, proposing a two-stage communication method to handle compressed key-value sequences that span rank boundaries, ensuring efficient and consistent compression across distributed ranks.", | |
| "text": "matrices. When the overall size of data parallelism exceeds the limit for ZeRO, we compute\nthe Muon update redundantly across the extra data-parallel groups, trading computation for\nreduced total bucket memory.\nFor MoE parameters, we optimize each expert independently. We first flatten all down\nprojection matrices in SwiGLU (Shazeer, 2020) of all experts across all layers, followed by\nflattened up projection matrices and gate matrices. Then, we pad the flattened vector to ensure\nwe can evenly distribute this vector across all ranks without splitting any logically independent\nmatrix. Given the large number of experts, we do not impose a limit of ZeRO parallelism for\nMoE parameters, and the padding overhead is also negligible.\nAdditionally, on each rank, consecutive parameters of identical shape will be automatically\nmerged, enabling batched execution of the Newton-Schulz iterations for better hardware utiliza-\ntion. Furthermore, we observe that the Newton-Schulz iterations in Muon remain stable when\ncomputed with BF16 matrix multiplications. Leveraging this, we further quantize, in a stochastic\nrounding manner, the MoE gradients to be synchronized across data-parallel ranks to the BF16\nprecision, halving the communication volume. To avoid accumulation errors introduced by\nlow-precision adders, we replace conventional tree- or ring-based reduce-scatter collectives with\na two-phase approach. First, an all-to-all operation exchanges local gradients across ranks, and\nthen each rank performs a local sum in FP32. This design maintains numerical robustness.\n3.4.2. Cost-Effective and Memory-Efficient Implementation of mHC\nThe introduction ofmHC increases both activation memory consumption and communication\nvolume between pipeline stages, compared with conventional residual connections. To mitigate\nthese costs, we implement several optimization strategies.\nFirstly, we carefully design and implement fused kernels ofmHC for both training and\ninference. Secondly, we introduce a recomputation strategy that selectively checkpoints interme-\ndiate tensors. Specifically, we recompute most hidden states between layers and all normalized\nlayer inputs, while avoiding recomputation of compute-intensive operations. This achieves a\nbalance between memory saving and computational overhead. Thirdly, we adjust the DualPipe\n1F1B overlapping scheme to accommodate the increased pipeline communication and enable\nconcurrent execution of some operations inmHC.\nCollectively, these optimizations constrain the wall-time overhead ofmHC to only 6.7% of\nthe overlapped 1F1B pipeline stage. More details of the engineering optimization can be found\nin the dedicatedmHC paper (Xie et al., 2026).\n3.4.3. Contextual Parallelism for Long-Context Attention\nConventional Context Parallelism (CP) partitions the sequence dimension, with each rank\nmaintaining contiguous \ud835\udc60tokens. This introduces two challenges to our compressed attention\nmechanisms (i.e., CSA and HCA). On the one hand, training samples are packed from multiple\nsequences, and each sequence is compressed independently by a factor of \ud835\udc5a(or\ud835\udc5a\u2032), with any\ntrailing tokens fewer than \ud835\udc5abeing discarded. Consequently, the compressed KV lengths are\ntypically less than\ud835\udc60\n\ud835\udc5aand vary across ranks. On the other hand, the compression requires \ud835\udc5a\nconsecutive KV entries, which may straddle the boundary between two neighboring CP ranks.\nTo address these challenges, we design a two-stage communication approach. In the first\nstage, each rank \ud835\udc56sends its last \ud835\udc5auncompressed KV entries to rank \ud835\udc56+1. Then, rank \ud835\udc56+1\ncompresses some of these received entries together with its local \ud835\udc60uncompressed KV entries,\n20" | |
| }, | |
| { | |
| "title": "Contextual Parallelism for Long-Context Attention", | |
| "node_id": "0018", | |
| "start_index": 20, | |
| "end_index": 21, | |
| "summary": "The partial document discusses advanced optimization techniques and implementation details for large-scale model training and inference, focusing on memory efficiency, parallelism, and computational trade-offs. Key points include:\n\n1. **Data Parallelism and ZeRO Limits:** When data parallelism exceeds ZeRO\u2019s capacity, redundant computation is used to reduce memory usage. For Mixture of Experts (MoE) parameters, experts are optimized independently by flattening and padding projection matrices to distribute evenly across ranks without splitting logical units, avoiding ZeRO limits and minimizing padding overhead.\n\n2. **Hardware Utilization and Precision Optimization:** Consecutive parameters of identical shape are merged to enable batched Newton-Schulz iterations, improving hardware efficiency. MoE gradients are quantized to BF16 precision with stochastic rounding to halve communication volume, while a two-phase gradient synchronization approach maintains numerical stability.\n\n3. **Memory and Communication Optimization for mHC:** The modified Highway Connection (mHC) increases memory and communication costs, addressed by fused kernel implementations, selective recomputation of intermediate tensors, and adjustments to pipeline overlapping schemes, limiting wall-time overhead to 6.7%.\n\n4. **Contextual Parallelism for Long-Context Attention:** Challenges in compressed attention mechanisms due to sequence partitioning are solved via a two-stage communication protocol involving exchange and compression of KV entries across ranks, followed by an all-gather and padding operation to reconstruct full compressed KV sets.\n\n5. **Extended Automatic Differentiation for Fine-Grained Activation Checkpointing:** A tensor-level checkpointing mechanism is introduced that integrates with automatic differentiation frameworks, allowing selective annotation of tensors for recomputation. This approach balances memory savings and computational overhead without manual backward implementation, leveraging graph tracing for minimal recomputation subgraphs and automatic deduplication.\n\n6. **Inference Framework and KV Cache Management:** The inference system builds on prior versions with enhancements in managing heterogeneous KV caches arising from hybrid attention mechanisms. A customized KV cache layout is designed to handle multiple KV entry types with varying sizes and update rules, including support for sparse selection via a lightning indexer.", | |
| "text": "matrices. When the overall size of data parallelism exceeds the limit for ZeRO, we compute\nthe Muon update redundantly across the extra data-parallel groups, trading computation for\nreduced total bucket memory.\nFor MoE parameters, we optimize each expert independently. We first flatten all down\nprojection matrices in SwiGLU (Shazeer, 2020) of all experts across all layers, followed by\nflattened up projection matrices and gate matrices. Then, we pad the flattened vector to ensure\nwe can evenly distribute this vector across all ranks without splitting any logically independent\nmatrix. Given the large number of experts, we do not impose a limit of ZeRO parallelism for\nMoE parameters, and the padding overhead is also negligible.\nAdditionally, on each rank, consecutive parameters of identical shape will be automatically\nmerged, enabling batched execution of the Newton-Schulz iterations for better hardware utiliza-\ntion. Furthermore, we observe that the Newton-Schulz iterations in Muon remain stable when\ncomputed with BF16 matrix multiplications. Leveraging this, we further quantize, in a stochastic\nrounding manner, the MoE gradients to be synchronized across data-parallel ranks to the BF16\nprecision, halving the communication volume. To avoid accumulation errors introduced by\nlow-precision adders, we replace conventional tree- or ring-based reduce-scatter collectives with\na two-phase approach. First, an all-to-all operation exchanges local gradients across ranks, and\nthen each rank performs a local sum in FP32. This design maintains numerical robustness.\n3.4.2. Cost-Effective and Memory-Efficient Implementation of mHC\nThe introduction ofmHC increases both activation memory consumption and communication\nvolume between pipeline stages, compared with conventional residual connections. To mitigate\nthese costs, we implement several optimization strategies.\nFirstly, we carefully design and implement fused kernels ofmHC for both training and\ninference. Secondly, we introduce a recomputation strategy that selectively checkpoints interme-\ndiate tensors. Specifically, we recompute most hidden states between layers and all normalized\nlayer inputs, while avoiding recomputation of compute-intensive operations. This achieves a\nbalance between memory saving and computational overhead. Thirdly, we adjust the DualPipe\n1F1B overlapping scheme to accommodate the increased pipeline communication and enable\nconcurrent execution of some operations inmHC.\nCollectively, these optimizations constrain the wall-time overhead ofmHC to only 6.7% of\nthe overlapped 1F1B pipeline stage. More details of the engineering optimization can be found\nin the dedicatedmHC paper (Xie et al., 2026).\n3.4.3. Contextual Parallelism for Long-Context Attention\nConventional Context Parallelism (CP) partitions the sequence dimension, with each rank\nmaintaining contiguous \ud835\udc60tokens. This introduces two challenges to our compressed attention\nmechanisms (i.e., CSA and HCA). On the one hand, training samples are packed from multiple\nsequences, and each sequence is compressed independently by a factor of \ud835\udc5a(or\ud835\udc5a\u2032), with any\ntrailing tokens fewer than \ud835\udc5abeing discarded. Consequently, the compressed KV lengths are\ntypically less than\ud835\udc60\n\ud835\udc5aand vary across ranks. On the other hand, the compression requires \ud835\udc5a\nconsecutive KV entries, which may straddle the boundary between two neighboring CP ranks.\nTo address these challenges, we design a two-stage communication approach. In the first\nstage, each rank \ud835\udc56sends its last \ud835\udc5auncompressed KV entries to rank \ud835\udc56+1. Then, rank \ud835\udc56+1\ncompresses some of these received entries together with its local \ud835\udc60uncompressed KV entries,\n20producing a fixed length of\ud835\udc60\n\ud835\udc5a+1 compressed entries, in which exist some padding entries. In\nthe second stage, an all-gather operation across all CP ranks collects the locally compressed KV\nentries. Then, a fused select-and-pad operator reorganizes them into the full set of compressed\nKV entries with a total length of cp_size\u00b7\ud835\udc60\n\ud835\udc5a. Any padding entries are placed at the tail. For\nHCA and the indexer in CSA, the visible range of compressed KV entries for each query token\ncan be precomputed by rules. For the sparse attention in CSA, the top- \ud835\udc58selector explicitly\nspecifies the indices of visible compressed KV entries for each query.\n3.4.4. Extended Automatic Differentiation for Flexible Activation Checkpointing\nConventional activation checkpointing implementations operate at the granularity of an entire\nmodule, deciding whether to retain or recompute its output activations during the backward\npass. This coarse granularity often leads to suboptimal trade-offs between recomputation cost\nand activation memory footprint. An alternative approach is to manually implement the forward\nand backward logic of an entire layer, explicitly managing tensor checkpointing states. While\nenabling fine-grained control, this method loses the convenience of the automatic differentiation\nframework, substantially increasing development complexity.\nTo achieve fine-grained control without sacrificing programming efficiency, we implement a\ntensor-level activation checkpointing mechanism with automatic differentiation support. With\nthis mechanism, developers only need to implement the forward pass and selectively annotate\nindividual tensors for automatic checkpointing and recomputation. Our framework leverages\nTorchFX (Reed et al., 2022) to trace the full computation graph. For each annotated tensor, it\nperforms a backward traversal to identify the minimal subgraph required for its recomputation.\nWe define these minimal subgraphs as recomputation graphs and insert them into the backward\nlogic just before the corresponding gradient computation.\nCompared with the manual implementation, this design introduces no additional overhead\nduring training. Recomputation in this framework is implemented by directly freeing the\nGPU memory of the annotated tensor and reusing the storage pointer from the recomputed\ntensor, without any GPU memory copy. Furthermore, since graph tracing executes the model\nconcretely, we can track the underlying storage pointer of each tensor, which enables automatic\ndeduplication of recomputation for tensors that share storage (e.g., the input and output of a\nreshape operation). This relieves developers from reasoning about low-level memory details\nwhen annotating recomputation.\n3.5. Inference Framework\nOur inference framework largely inherits from that of DeepSeek-V3, with some differences in\nKV Cache management.\n3.5.1. KV Cache Structure and Management\nTo efficiently manage the heterogeneous KV caches arising from the hybrid attention mechanism\nin DeepSeek-V4, we design a customized KV cache layout. The layout is illustrated in Figure 6,\nand we will elaborate on it in detail as follows.\nHeterogeneous KV Entries in DeepSeek-V4.The hybrid attention mechanism in DeepSeek-\nV4 series introduces multiple types of KV entries with different Key-Value (KV) cache sizes\nand update rules. The lightning indexer for sparse selection introduces additional dimensions\n21" | |
| }, | |
| { | |
| "title": "Extended Automatic Differentiation for Flexible Activation Checkpointing", | |
| "node_id": "0019", | |
| "start_index": 21, | |
| "end_index": 21, | |
| "summary": "The partial document discusses advanced techniques for managing compressed Key-Value (KV) entries and activation checkpointing in a deep learning framework. It details a two-stage process for compressing and gathering KV entries across compute ranks, including padding strategies and selection mechanisms for sparse attention. The document then introduces an extended automatic differentiation approach for fine-grained, tensor-level activation checkpointing that balances memory efficiency and computational overhead by leveraging TorchFX for computation graph tracing and minimal recomputation subgraphs. This method automates memory management and reduces developer complexity compared to manual implementations. Finally, the document outlines the inference framework, focusing on a customized KV cache layout designed to handle heterogeneous KV entries generated by a hybrid attention mechanism in DeepSeek-V4, addressing different cache sizes and update rules for efficient KV cache management.", | |
| "text": "producing a fixed length of\ud835\udc60\n\ud835\udc5a+1 compressed entries, in which exist some padding entries. In\nthe second stage, an all-gather operation across all CP ranks collects the locally compressed KV\nentries. Then, a fused select-and-pad operator reorganizes them into the full set of compressed\nKV entries with a total length of cp_size\u00b7\ud835\udc60\n\ud835\udc5a. Any padding entries are placed at the tail. For\nHCA and the indexer in CSA, the visible range of compressed KV entries for each query token\ncan be precomputed by rules. For the sparse attention in CSA, the top- \ud835\udc58selector explicitly\nspecifies the indices of visible compressed KV entries for each query.\n3.4.4. Extended Automatic Differentiation for Flexible Activation Checkpointing\nConventional activation checkpointing implementations operate at the granularity of an entire\nmodule, deciding whether to retain or recompute its output activations during the backward\npass. This coarse granularity often leads to suboptimal trade-offs between recomputation cost\nand activation memory footprint. An alternative approach is to manually implement the forward\nand backward logic of an entire layer, explicitly managing tensor checkpointing states. While\nenabling fine-grained control, this method loses the convenience of the automatic differentiation\nframework, substantially increasing development complexity.\nTo achieve fine-grained control without sacrificing programming efficiency, we implement a\ntensor-level activation checkpointing mechanism with automatic differentiation support. With\nthis mechanism, developers only need to implement the forward pass and selectively annotate\nindividual tensors for automatic checkpointing and recomputation. Our framework leverages\nTorchFX (Reed et al., 2022) to trace the full computation graph. For each annotated tensor, it\nperforms a backward traversal to identify the minimal subgraph required for its recomputation.\nWe define these minimal subgraphs as recomputation graphs and insert them into the backward\nlogic just before the corresponding gradient computation.\nCompared with the manual implementation, this design introduces no additional overhead\nduring training. Recomputation in this framework is implemented by directly freeing the\nGPU memory of the annotated tensor and reusing the storage pointer from the recomputed\ntensor, without any GPU memory copy. Furthermore, since graph tracing executes the model\nconcretely, we can track the underlying storage pointer of each tensor, which enables automatic\ndeduplication of recomputation for tensors that share storage (e.g., the input and output of a\nreshape operation). This relieves developers from reasoning about low-level memory details\nwhen annotating recomputation.\n3.5. Inference Framework\nOur inference framework largely inherits from that of DeepSeek-V3, with some differences in\nKV Cache management.\n3.5.1. KV Cache Structure and Management\nTo efficiently manage the heterogeneous KV caches arising from the hybrid attention mechanism\nin DeepSeek-V4, we design a customized KV cache layout. The layout is illustrated in Figure 6,\nand we will elaborate on it in detail as follows.\nHeterogeneous KV Entries in DeepSeek-V4.The hybrid attention mechanism in DeepSeek-\nV4 series introduces multiple types of KV entries with different Key-Value (KV) cache sizes\nand update rules. The lightning indexer for sparse selection introduces additional dimensions\n21" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Inference Framework", | |
| "node_id": "0020", | |
| "start_index": 21, | |
| "end_index": 21, | |
| "summary": "The partial document discusses advanced techniques for managing compressed Key-Value (KV) entries and activation checkpointing in a deep learning framework. It details a two-stage process for compressing and gathering KV entries across compute ranks, including padding strategies and selection mechanisms for sparse attention. The document then introduces an extended automatic differentiation approach for fine-grained, tensor-level activation checkpointing that balances memory efficiency and computational overhead without sacrificing programming convenience. This method leverages TorchFX to trace computation graphs and automatically manage recomputation subgraphs, optimizing GPU memory usage. Finally, the document outlines the inference framework, focusing on a customized KV cache layout designed to handle heterogeneous KV entries generated by a hybrid attention mechanism in DeepSeek-V4, addressing the complexity of varying cache sizes and update rules.", | |
| "text": "producing a fixed length of\ud835\udc60\n\ud835\udc5a+1 compressed entries, in which exist some padding entries. In\nthe second stage, an all-gather operation across all CP ranks collects the locally compressed KV\nentries. Then, a fused select-and-pad operator reorganizes them into the full set of compressed\nKV entries with a total length of cp_size\u00b7\ud835\udc60\n\ud835\udc5a. Any padding entries are placed at the tail. For\nHCA and the indexer in CSA, the visible range of compressed KV entries for each query token\ncan be precomputed by rules. For the sparse attention in CSA, the top- \ud835\udc58selector explicitly\nspecifies the indices of visible compressed KV entries for each query.\n3.4.4. Extended Automatic Differentiation for Flexible Activation Checkpointing\nConventional activation checkpointing implementations operate at the granularity of an entire\nmodule, deciding whether to retain or recompute its output activations during the backward\npass. This coarse granularity often leads to suboptimal trade-offs between recomputation cost\nand activation memory footprint. An alternative approach is to manually implement the forward\nand backward logic of an entire layer, explicitly managing tensor checkpointing states. While\nenabling fine-grained control, this method loses the convenience of the automatic differentiation\nframework, substantially increasing development complexity.\nTo achieve fine-grained control without sacrificing programming efficiency, we implement a\ntensor-level activation checkpointing mechanism with automatic differentiation support. With\nthis mechanism, developers only need to implement the forward pass and selectively annotate\nindividual tensors for automatic checkpointing and recomputation. Our framework leverages\nTorchFX (Reed et al., 2022) to trace the full computation graph. For each annotated tensor, it\nperforms a backward traversal to identify the minimal subgraph required for its recomputation.\nWe define these minimal subgraphs as recomputation graphs and insert them into the backward\nlogic just before the corresponding gradient computation.\nCompared with the manual implementation, this design introduces no additional overhead\nduring training. Recomputation in this framework is implemented by directly freeing the\nGPU memory of the annotated tensor and reusing the storage pointer from the recomputed\ntensor, without any GPU memory copy. Furthermore, since graph tracing executes the model\nconcretely, we can track the underlying storage pointer of each tensor, which enables automatic\ndeduplication of recomputation for tensors that share storage (e.g., the input and output of a\nreshape operation). This relieves developers from reasoning about low-level memory details\nwhen annotating recomputation.\n3.5. Inference Framework\nOur inference framework largely inherits from that of DeepSeek-V3, with some differences in\nKV Cache management.\n3.5.1. KV Cache Structure and Management\nTo efficiently manage the heterogeneous KV caches arising from the hybrid attention mechanism\nin DeepSeek-V4, we design a customized KV cache layout. The layout is illustrated in Figure 6,\nand we will elaborate on it in detail as follows.\nHeterogeneous KV Entries in DeepSeek-V4.The hybrid attention mechanism in DeepSeek-\nV4 series introduces multiple types of KV entries with different Key-Value (KV) cache sizes\nand update rules. The lightning indexer for sparse selection introduces additional dimensions\n21", | |
| "nodes": [ | |
| { | |
| "title": "KV Cache Structure and Management", | |
| "node_id": "0021", | |
| "start_index": 21, | |
| "end_index": 23, | |
| "summary": "The partial document discusses advanced techniques for managing compressed key-value (KV) caches and activation checkpointing in a hybrid attention model, specifically DeepSeek-V4. It covers a multi-stage process for compressing and gathering KV entries, including padding and reorganization strategies for different attention mechanisms (CSA, HCA, SWA). The document introduces an extended automatic differentiation framework enabling fine-grained, tensor-level activation checkpointing that balances memory savings and recomputation cost without sacrificing programming convenience or training efficiency. It details a customized KV cache layout designed to handle heterogeneous KV entries arising from hybrid attention, addressing challenges such as diverse cache policies and alignment constraints through state caches and co-designed sparse attention kernels. Additionally, it presents an on-disk KV cache storage system to optimize inference by avoiding redundant computations for shared-prefix requests, proposing three strategies (full caching, periodic checkpointing, zero caching) to manage the trade-offs between storage overhead and computational redundancy, particularly for large uncompressed SWA KV entries.", | |
| "text": "producing a fixed length of\ud835\udc60\n\ud835\udc5a+1 compressed entries, in which exist some padding entries. In\nthe second stage, an all-gather operation across all CP ranks collects the locally compressed KV\nentries. Then, a fused select-and-pad operator reorganizes them into the full set of compressed\nKV entries with a total length of cp_size\u00b7\ud835\udc60\n\ud835\udc5a. Any padding entries are placed at the tail. For\nHCA and the indexer in CSA, the visible range of compressed KV entries for each query token\ncan be precomputed by rules. For the sparse attention in CSA, the top- \ud835\udc58selector explicitly\nspecifies the indices of visible compressed KV entries for each query.\n3.4.4. Extended Automatic Differentiation for Flexible Activation Checkpointing\nConventional activation checkpointing implementations operate at the granularity of an entire\nmodule, deciding whether to retain or recompute its output activations during the backward\npass. This coarse granularity often leads to suboptimal trade-offs between recomputation cost\nand activation memory footprint. An alternative approach is to manually implement the forward\nand backward logic of an entire layer, explicitly managing tensor checkpointing states. While\nenabling fine-grained control, this method loses the convenience of the automatic differentiation\nframework, substantially increasing development complexity.\nTo achieve fine-grained control without sacrificing programming efficiency, we implement a\ntensor-level activation checkpointing mechanism with automatic differentiation support. With\nthis mechanism, developers only need to implement the forward pass and selectively annotate\nindividual tensors for automatic checkpointing and recomputation. Our framework leverages\nTorchFX (Reed et al., 2022) to trace the full computation graph. For each annotated tensor, it\nperforms a backward traversal to identify the minimal subgraph required for its recomputation.\nWe define these minimal subgraphs as recomputation graphs and insert them into the backward\nlogic just before the corresponding gradient computation.\nCompared with the manual implementation, this design introduces no additional overhead\nduring training. Recomputation in this framework is implemented by directly freeing the\nGPU memory of the annotated tensor and reusing the storage pointer from the recomputed\ntensor, without any GPU memory copy. Furthermore, since graph tracing executes the model\nconcretely, we can track the underlying storage pointer of each tensor, which enables automatic\ndeduplication of recomputation for tensors that share storage (e.g., the input and output of a\nreshape operation). This relieves developers from reasoning about low-level memory details\nwhen annotating recomputation.\n3.5. Inference Framework\nOur inference framework largely inherits from that of DeepSeek-V3, with some differences in\nKV Cache management.\n3.5.1. KV Cache Structure and Management\nTo efficiently manage the heterogeneous KV caches arising from the hybrid attention mechanism\nin DeepSeek-V4, we design a customized KV cache layout. The layout is illustrated in Figure 6,\nand we will elaborate on it in detail as follows.\nHeterogeneous KV Entries in DeepSeek-V4.The hybrid attention mechanism in DeepSeek-\nV4 series introduces multiple types of KV entries with different Key-Value (KV) cache sizes\nand update rules. The lightning indexer for sparse selection introduces additional dimensions\n21State Cache\nSWA KVKV Cache\nBlock 0\nBlock 1\nBlock 2\nBlock NSWA KV\nSWA KV\nSWA KV\nSWA KVUncompressed\nKV State\nUncompressed\nKV StateUncompressed\nKV State\nUncompressed\nKV State\nUncompressed\nKV StateLayer-2 CSA State\nLayer-3 HCA State\n\u2026Layer-0 SW A KV\n\u2026\nLayer-n SW A KVRequest 1\nRequest RRequest 2\nRequest 3\u2026\u2026CSA KV\nHCA KV\nCSA KV\nHCA KV\nCSA KV\nHCA KV\nCSA KV\nHCA KV\nCSA KV\nHCA KVCSA Indexer KV\n of k1 tokensCSA Main KV\nof k1 tokens\nHCA KV of k2 tokens\nCSA Indexer KV\n of k1 tokensCSA Main KV\nof k1 tokens\nHCA KV of k2 tokensLayer-2\nLayer-3\nLayer-5Layer-4\n... ...\n\u2026\u2026Figure 6|Illustration of the KV cache Layout for DeepSeek-V4. The KV cache is organized into\ntwo primary components: a classical KV cache for CSA/HCA, and a state cache for SWA and\nunready-for-compression tokens in CSA/HCA. In the state cache, each request is assigned a\nfixed-size cache block. Within this block, the SWA segment stores the KV entries corresponding\nto the most recent \ud835\udc5bwintokens, while the CSA/HCA segment stores uncompressed tail states\nthat are not yet ready for compression. In the classical KV cache, we allocate multiple blocks\nper request. Each cache block covers lcm(\ud835\udc5a ,\ud835\udc5a\u2032)original tokens, producing \ud835\udc581=lcm(\ud835\udc5a,\ud835\udc5a\u2032)\n\ud835\udc5aCSA\ncompressed tokens and\ud835\udc58 2=lcm(\ud835\udc5a,\ud835\udc5a\u2032)\n\ud835\udc5a\u2032 HCA compressed tokens.\ninto the KV cache that possess embedding sizes distinct from those in the primary attention.\nThe compression techniques employed in CSA and HCA reduce the sequence length by factors\nof1\n\ud835\udc5aand1\n\ud835\udc5a\u2032, respectively, thereby decreasing the overall KV cache size. As a result, KV cache\nsizes vary across different layers. Furthermore, Sliding Window Attention (SWA) layers also\noperate with distinct KV cache sizes, as well as separate cache hit and eviction policies. In\nthe compression branch, one KV entry is generated for every \ud835\udc5atokens. When the number\nof remaining tokens is insufficient for compression, all pending tokens and their associated\nhidden states must be retained in a buffer until the compression operation can be executed.\nThese buffered tokens represent a sequence state determined by positional context and are also\nmanaged within the KV cache framework.\nChallenges in Managing Hybrid Attention KV Cache.The hybrid attention mechanism\nviolates fundamental assumptions behind PagedAttention and its variants. Although recent\nhybrid KV cache managing algorithms (e.g., Jenga (Zhang et al., 2025a), Hymba (Dong et al.,\n2025)) target general hybrid attention models or specific structures, two principal obstacles\nprevent consolidating KV caches across all layers under the PagedAttention framework:\n\u2022 Diverse cache policies, such as those used in Sliding Window Attention.\n\u2022Constraints imposed by high-performance attention kernels, including alignment require-\nments.\nFor efficient KV cache management of DeepSeek-V4, we design corresponding strategies to\novercome these two challenges.\nState Cache for SWA and Uncompressed Tail Tokens.To address the first obstacle, we adopt\nan alternative cache management mechanism. Since SWA is designed to enhance performance\nunder a limited KV cache size, it is reasonable to treat it, along with the uncompressed tail tokens\nfrom the compression branch, as a state-space model. The corresponding KV cache can thus be\n22regarded as a sequence-specific state that depends solely on the current position. Accordingly,\nwe pre-allocate a fixed- and limited-size pool of state caches, and dynamically assign it to each\nsequence.\nSparse Attention Kernel Co-Design.Regarding the second obstacle, conventional high-\nperformance attention kernels typically assume a fixed number \ud835\udc35of tokens per block to optimize\nperformance, corresponding to \ud835\udc35\u00b7\ud835\udc5a original tokens in CSA and \ud835\udc35\u00b7\ud835\udc5a\u2032in HCA. Through em-\nploying a high-performance sparse-attention kernel, different layers can accommodate variable\ntokens per block without performance degradation. Achieving this requires co-designing the\nKV cache layout and the sparse attention kernel. For instance, padding blocks to align with\ncache lines can improve performance. Thus, for CSA with compression ratio \ud835\udc5aand HCA with\nratio\ud835\udc5a\u2032, the number of original tokens per block can be any multiple of lcm(\ud835\udc5a ,\ud835\udc5a\u2032), the least\ncommon multiple of these two compression ratios.\n3.5.2. On-Disk KV Cache Storage\nWhen serving DeepSeek-V4, we leverage an on-disk KV cache storage mechanism to eliminate\nrepeated prefilling for shared-prefix requests. For the compressed KV entries in CSA/HCA and\nthe uncompressed KV entries in Sliding Window Attention (SWA), we design separate solutions\nfor storage management.\nFor CSA and HCA, we simply store all of the compressed KV entries to the disk. When\na request hits a stored prefix, we read and reuse the compressed KV entries corresponding\nto the prefix, until the last complete compression block. Specially, for prefix tokens in the tail\nincomplete block, we still need to recompute them to restore the uncompressed KV entries, as\nuncompressed KV entries in CSA and HCA are not stored.\nFor the SWA KV entries, since they are not compressed and exist in every layer, their volume\nis approximately 8 times larger than the compressed CSA and HCA KV entries. To handle\nthese large SWA KV entries efficiently, we propose and implement three distinct strategies for\nmanaging on-disk SWA KV entries, each offering a different trade-off between storage overhead\nand computational redundancy:\n\u2022Full SWA Caching.This strategy stores the complete SWA KV entries for all tokens,\nensuring computational zero-redundancy. Under this strategy, the SWA KV entries of the\nhitting prefix can be reconstructed by just reading the on-disk cache of the last \ud835\udc5bwintokens\nwithin that prefix. Despite computational zero-redundancy, this strategy is inefficient for\nmodern SSD-based storage systems \u2014 only a small subset of the stored SWA KV cache\nwill be accessed for each hitting request, which leads to an unbalanced write-intensive\naccess pattern.\n\u2022Periodic Checkpointing.This strategy checkpoints SWA KV entries of the last \ud835\udc5bwintokens\nwithin every \ud835\udc5dtokens, where \ud835\udc5dis a tunable parameter. For a hitting prefix, we load the\nmost recent checkpointed state, and then recompute the remaining tail tokens. Through\ntuning\ud835\udc5d, this strategy enables an on-demand trade-off between storage and computation.\n\u2022Zero SWA Caching.This strategy does not store any SWA KV entries. For a hitting prefix,\nwe need to perform more recomputation to restore the SWA KV entries. To be specific, in\neach attention layer, the SWA KV entry of each token depends on the SWA KV entries of\nonly the most recent \ud835\udc5bwintokens from the previous layer. Therefore, leveraging cached\nCSA and HCA KV entries, recomputing the last \ud835\udc5bwin\u00b7\ud835\udc3ftokens is enough to restore the last\n\ud835\udc5bwinSWA KV entries for an\ud835\udc3f-layer model.\n23" | |
| }, | |
| { | |
| "title": "On-Disk KV Cache Storage", | |
| "node_id": "0022", | |
| "start_index": 23, | |
| "end_index": 24, | |
| "summary": "The partial document discusses advanced techniques for efficient attention mechanisms and key-value (KV) cache management in large-scale transformer models, specifically focusing on DeepSeek-V4. It covers the design and optimization of sparse attention kernels that support variable token blocks without performance loss by co-designing KV cache layouts and attention kernels. The document details an on-disk KV cache storage system to avoid redundant computations for shared-prefix requests, describing distinct strategies for managing compressed (CSA/HCA) and uncompressed (SWA) KV entries, each balancing storage overhead and computational redundancy differently.\n\nAdditionally, the document outlines the pre-training data construction for DeepSeek-V4, emphasizing a diverse, high-quality corpus with longer contexts, including filtered web data, mathematical and programming content, multilingual sources, and long documents like scientific papers. It also describes tokenization and pre-processing strategies inherited and improved from DeepSeek-V3.\n\nFinally, the document presents the model architecture and training setup for DeepSeek-V4-Flash, detailing the number of layers, attention mechanisms (sliding window, CSA, HCA), compression rates, MoE layer configurations, and parameter counts, highlighting the model\u2019s scale and complexity.", | |
| "text": "regarded as a sequence-specific state that depends solely on the current position. Accordingly,\nwe pre-allocate a fixed- and limited-size pool of state caches, and dynamically assign it to each\nsequence.\nSparse Attention Kernel Co-Design.Regarding the second obstacle, conventional high-\nperformance attention kernels typically assume a fixed number \ud835\udc35of tokens per block to optimize\nperformance, corresponding to \ud835\udc35\u00b7\ud835\udc5a original tokens in CSA and \ud835\udc35\u00b7\ud835\udc5a\u2032in HCA. Through em-\nploying a high-performance sparse-attention kernel, different layers can accommodate variable\ntokens per block without performance degradation. Achieving this requires co-designing the\nKV cache layout and the sparse attention kernel. For instance, padding blocks to align with\ncache lines can improve performance. Thus, for CSA with compression ratio \ud835\udc5aand HCA with\nratio\ud835\udc5a\u2032, the number of original tokens per block can be any multiple of lcm(\ud835\udc5a ,\ud835\udc5a\u2032), the least\ncommon multiple of these two compression ratios.\n3.5.2. On-Disk KV Cache Storage\nWhen serving DeepSeek-V4, we leverage an on-disk KV cache storage mechanism to eliminate\nrepeated prefilling for shared-prefix requests. For the compressed KV entries in CSA/HCA and\nthe uncompressed KV entries in Sliding Window Attention (SWA), we design separate solutions\nfor storage management.\nFor CSA and HCA, we simply store all of the compressed KV entries to the disk. When\na request hits a stored prefix, we read and reuse the compressed KV entries corresponding\nto the prefix, until the last complete compression block. Specially, for prefix tokens in the tail\nincomplete block, we still need to recompute them to restore the uncompressed KV entries, as\nuncompressed KV entries in CSA and HCA are not stored.\nFor the SWA KV entries, since they are not compressed and exist in every layer, their volume\nis approximately 8 times larger than the compressed CSA and HCA KV entries. To handle\nthese large SWA KV entries efficiently, we propose and implement three distinct strategies for\nmanaging on-disk SWA KV entries, each offering a different trade-off between storage overhead\nand computational redundancy:\n\u2022Full SWA Caching.This strategy stores the complete SWA KV entries for all tokens,\nensuring computational zero-redundancy. Under this strategy, the SWA KV entries of the\nhitting prefix can be reconstructed by just reading the on-disk cache of the last \ud835\udc5bwintokens\nwithin that prefix. Despite computational zero-redundancy, this strategy is inefficient for\nmodern SSD-based storage systems \u2014 only a small subset of the stored SWA KV cache\nwill be accessed for each hitting request, which leads to an unbalanced write-intensive\naccess pattern.\n\u2022Periodic Checkpointing.This strategy checkpoints SWA KV entries of the last \ud835\udc5bwintokens\nwithin every \ud835\udc5dtokens, where \ud835\udc5dis a tunable parameter. For a hitting prefix, we load the\nmost recent checkpointed state, and then recompute the remaining tail tokens. Through\ntuning\ud835\udc5d, this strategy enables an on-demand trade-off between storage and computation.\n\u2022Zero SWA Caching.This strategy does not store any SWA KV entries. For a hitting prefix,\nwe need to perform more recomputation to restore the SWA KV entries. To be specific, in\neach attention layer, the SWA KV entry of each token depends on the SWA KV entries of\nonly the most recent \ud835\udc5bwintokens from the previous layer. Therefore, leveraging cached\nCSA and HCA KV entries, recomputing the last \ud835\udc5bwin\u00b7\ud835\udc3ftokens is enough to restore the last\n\ud835\udc5bwinSWA KV entries for an\ud835\udc3f-layer model.\n23Depending on specific deployment scenarios, we select the most suitable strategy to achieve the\ndesired trade-off between storage and computation.\n4. Pre-Training\n4.1. Data Construction\nOn top of the pre-training data of DeepSeek-V3, we endeavor to construct a more diverse and\nhigher-quality training corpus with longer effective contexts. We continually refine our data con-\nstruction pipelines. For web-sourced data, we implement filtering strategies to remove batched\nauto-generated and templated content, thereby mitigating the risk of model collapse (Zhu et al.,\n2024). Mathematical and programming corpora still remain core components of our training\ndata, and we further enhance the coding capabilities of DeepSeek-V4 series by incorporating\nagentic data during the mid-training phase. For multilingual data, we build a larger corpus\nfor DeepSeek-V4, improving its capture of long-tail knowledge across different cultures. For\nDeepSeek-V4, we place a particular emphasis on long-document data curation, prioritizing\nscientific papers, technical reports, and other materials that reflect unique academic values.\nCombining all the above, our pre-training corpus comprises more than 32T tokens, containing\nmathematical contents, codes, web pages, long documents, and other high-quality categories.\nFor pre-training data, we largely follow the same pre-processing strategies of DeepSeek-\nV3. For tokenization, on top of the DeepSeek-V3 tokenizer, we introduce a few special tokens\nfor context construction, and still remain the vocabulary size to be 128K. We also inherit the\ntoken-splitting (DeepSeek-AI, 2024) and Fill-in-Middle (FIM) (DeepSeek-AI, 2024) strategies\nfrom DeepSeek-V3. Inspired by Ding et al. (2024), we pack documents from different sources\ninto appropriate sequences to minimize sample truncation. Different from DeepSeek-V3, we\nemploy sample-level attention masking during pre-training.\n4.2. Pre-Training Setups\n4.2.1. Model Setups\nDeepSeek-V4-Flash.We set the number of Transformer layers to 43 and the hidden dimension\n\ud835\udc51to 4096. For the first two layers, we use pure sliding window attention. For the subsequent\nlayers, CSA and HCA are used in an interleaved manner. For CSA, we set the compression rate\n\ud835\udc5ato 4, the number of indexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and\nthe number of KV entries selected for sparse attention (i.e., attention top-k) to 512. For HCA,\nwe set the compression rate \ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query\nheads\ud835\udc5b\u210eto 64, the head dimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1024.\nThe number of output projection groups \ud835\udc54is set to 8, and the dimension of each intermediate\nattention output \ud835\udc51\ud835\udc54is set to 1024. For the additional branch of sliding window attention, the\nwindow size \ud835\udc5bwinis set to 128. We employ MoE layers in all Transformer blocks, but use the\nHash routing strategy for the first 3 MoE layers. Each MoE layer consists of 1 shared expert and\n256 routed experts, where the intermediate hidden dimension of each expert is 2048. Among the\nrouted experts, 6 experts will be activated for each token. The multi-token prediction depth is\nset to 1. As formHC, the expansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp\niterations \ud835\udc61maxis set to 20. Under this configuration, DeepSeek-V4-Flash comprises 284B total\nparameters, of which 13B are activated for each token.\n24" | |
| } | |
| ] | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Pre-Training", | |
| "node_id": "0023", | |
| "start_index": 24, | |
| "end_index": 24, | |
| "summary": "The partial document primarily covers the strategies and methodologies employed in the pre-training phase of the DeepSeek-V4 model series. It details the construction of a diverse and high-quality training corpus, emphasizing the inclusion of long-context data, mathematical and programming content, multilingual sources, and long documents such as scientific papers and technical reports. The document also describes the data preprocessing and tokenization techniques, including enhancements over the previous DeepSeek-V3 version, such as special tokens, token-splitting, Fill-in-Middle strategies, and sample-level attention masking. Additionally, it outlines the specific architectural and hyperparameter configurations of the DeepSeek-V4-Flash model, including the number of Transformer layers, attention mechanisms (sliding window, CSA, HCA), MoE layers with Hash routing, and detailed parameter settings, highlighting the model\u2019s scale and complexity aimed at balancing storage and computation trade-offs.", | |
| "text": "Depending on specific deployment scenarios, we select the most suitable strategy to achieve the\ndesired trade-off between storage and computation.\n4. Pre-Training\n4.1. Data Construction\nOn top of the pre-training data of DeepSeek-V3, we endeavor to construct a more diverse and\nhigher-quality training corpus with longer effective contexts. We continually refine our data con-\nstruction pipelines. For web-sourced data, we implement filtering strategies to remove batched\nauto-generated and templated content, thereby mitigating the risk of model collapse (Zhu et al.,\n2024). Mathematical and programming corpora still remain core components of our training\ndata, and we further enhance the coding capabilities of DeepSeek-V4 series by incorporating\nagentic data during the mid-training phase. For multilingual data, we build a larger corpus\nfor DeepSeek-V4, improving its capture of long-tail knowledge across different cultures. For\nDeepSeek-V4, we place a particular emphasis on long-document data curation, prioritizing\nscientific papers, technical reports, and other materials that reflect unique academic values.\nCombining all the above, our pre-training corpus comprises more than 32T tokens, containing\nmathematical contents, codes, web pages, long documents, and other high-quality categories.\nFor pre-training data, we largely follow the same pre-processing strategies of DeepSeek-\nV3. For tokenization, on top of the DeepSeek-V3 tokenizer, we introduce a few special tokens\nfor context construction, and still remain the vocabulary size to be 128K. We also inherit the\ntoken-splitting (DeepSeek-AI, 2024) and Fill-in-Middle (FIM) (DeepSeek-AI, 2024) strategies\nfrom DeepSeek-V3. Inspired by Ding et al. (2024), we pack documents from different sources\ninto appropriate sequences to minimize sample truncation. Different from DeepSeek-V3, we\nemploy sample-level attention masking during pre-training.\n4.2. Pre-Training Setups\n4.2.1. Model Setups\nDeepSeek-V4-Flash.We set the number of Transformer layers to 43 and the hidden dimension\n\ud835\udc51to 4096. For the first two layers, we use pure sliding window attention. For the subsequent\nlayers, CSA and HCA are used in an interleaved manner. For CSA, we set the compression rate\n\ud835\udc5ato 4, the number of indexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and\nthe number of KV entries selected for sparse attention (i.e., attention top-k) to 512. For HCA,\nwe set the compression rate \ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query\nheads\ud835\udc5b\u210eto 64, the head dimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1024.\nThe number of output projection groups \ud835\udc54is set to 8, and the dimension of each intermediate\nattention output \ud835\udc51\ud835\udc54is set to 1024. For the additional branch of sliding window attention, the\nwindow size \ud835\udc5bwinis set to 128. We employ MoE layers in all Transformer blocks, but use the\nHash routing strategy for the first 3 MoE layers. Each MoE layer consists of 1 shared expert and\n256 routed experts, where the intermediate hidden dimension of each expert is 2048. Among the\nrouted experts, 6 experts will be activated for each token. The multi-token prediction depth is\nset to 1. As formHC, the expansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp\niterations \ud835\udc61maxis set to 20. Under this configuration, DeepSeek-V4-Flash comprises 284B total\nparameters, of which 13B are activated for each token.\n24", | |
| "nodes": [ | |
| { | |
| "title": "Data Construction", | |
| "node_id": "0024", | |
| "start_index": 24, | |
| "end_index": 24, | |
| "summary": "The partial document primarily covers the pre-training phase of the DeepSeek-V4 model series. It details the construction of a diverse and high-quality training corpus that includes mathematical content, programming code, multilingual data, and long documents such as scientific papers and technical reports, totaling over 32 trillion tokens. The document explains the data filtering and preprocessing strategies, including tokenization enhancements and attention masking techniques inherited and improved from DeepSeek-V3. Additionally, it describes the specific model architecture and training setup for DeepSeek-V4-Flash, outlining the Transformer layer configurations, attention mechanisms (sliding window, CSA, HCA), Mixture of Experts (MoE) layers with Hash routing, and various hyperparameters, resulting in a model with 284 billion parameters and optimized token activation.", | |
| "text": "Depending on specific deployment scenarios, we select the most suitable strategy to achieve the\ndesired trade-off between storage and computation.\n4. Pre-Training\n4.1. Data Construction\nOn top of the pre-training data of DeepSeek-V3, we endeavor to construct a more diverse and\nhigher-quality training corpus with longer effective contexts. We continually refine our data con-\nstruction pipelines. For web-sourced data, we implement filtering strategies to remove batched\nauto-generated and templated content, thereby mitigating the risk of model collapse (Zhu et al.,\n2024). Mathematical and programming corpora still remain core components of our training\ndata, and we further enhance the coding capabilities of DeepSeek-V4 series by incorporating\nagentic data during the mid-training phase. For multilingual data, we build a larger corpus\nfor DeepSeek-V4, improving its capture of long-tail knowledge across different cultures. For\nDeepSeek-V4, we place a particular emphasis on long-document data curation, prioritizing\nscientific papers, technical reports, and other materials that reflect unique academic values.\nCombining all the above, our pre-training corpus comprises more than 32T tokens, containing\nmathematical contents, codes, web pages, long documents, and other high-quality categories.\nFor pre-training data, we largely follow the same pre-processing strategies of DeepSeek-\nV3. For tokenization, on top of the DeepSeek-V3 tokenizer, we introduce a few special tokens\nfor context construction, and still remain the vocabulary size to be 128K. We also inherit the\ntoken-splitting (DeepSeek-AI, 2024) and Fill-in-Middle (FIM) (DeepSeek-AI, 2024) strategies\nfrom DeepSeek-V3. Inspired by Ding et al. (2024), we pack documents from different sources\ninto appropriate sequences to minimize sample truncation. Different from DeepSeek-V3, we\nemploy sample-level attention masking during pre-training.\n4.2. Pre-Training Setups\n4.2.1. Model Setups\nDeepSeek-V4-Flash.We set the number of Transformer layers to 43 and the hidden dimension\n\ud835\udc51to 4096. For the first two layers, we use pure sliding window attention. For the subsequent\nlayers, CSA and HCA are used in an interleaved manner. For CSA, we set the compression rate\n\ud835\udc5ato 4, the number of indexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and\nthe number of KV entries selected for sparse attention (i.e., attention top-k) to 512. For HCA,\nwe set the compression rate \ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query\nheads\ud835\udc5b\u210eto 64, the head dimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1024.\nThe number of output projection groups \ud835\udc54is set to 8, and the dimension of each intermediate\nattention output \ud835\udc51\ud835\udc54is set to 1024. For the additional branch of sliding window attention, the\nwindow size \ud835\udc5bwinis set to 128. We employ MoE layers in all Transformer blocks, but use the\nHash routing strategy for the first 3 MoE layers. Each MoE layer consists of 1 shared expert and\n256 routed experts, where the intermediate hidden dimension of each expert is 2048. Among the\nrouted experts, 6 experts will be activated for each token. The multi-token prediction depth is\nset to 1. As formHC, the expansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp\niterations \ud835\udc61maxis set to 20. Under this configuration, DeepSeek-V4-Flash comprises 284B total\nparameters, of which 13B are activated for each token.\n24" | |
| }, | |
| { | |
| "title": "Pre-Training Setups", | |
| "node_id": "0025", | |
| "start_index": 24, | |
| "end_index": 24, | |
| "summary": "The partial document focuses on the pre-training phase of the DeepSeek-V4 model series, detailing the data construction and model setup strategies. It describes efforts to build a diverse, high-quality training corpus exceeding 32 trillion tokens, emphasizing long-context data, multilingual content, and academic materials such as scientific papers and technical reports. The document outlines improvements over the previous DeepSeek-V3, including enhanced filtering of web data, incorporation of agentic data for coding capabilities, and specialized tokenization techniques. It also explains the architectural configuration of the DeepSeek-V4-Flash model, specifying the number of Transformer layers, attention mechanisms (sliding window, CSA, HCA), MoE layers with Hash routing, and various hyperparameters, culminating in a 284 billion parameter model optimized for efficient computation and storage trade-offs.", | |
| "text": "Depending on specific deployment scenarios, we select the most suitable strategy to achieve the\ndesired trade-off between storage and computation.\n4. Pre-Training\n4.1. Data Construction\nOn top of the pre-training data of DeepSeek-V3, we endeavor to construct a more diverse and\nhigher-quality training corpus with longer effective contexts. We continually refine our data con-\nstruction pipelines. For web-sourced data, we implement filtering strategies to remove batched\nauto-generated and templated content, thereby mitigating the risk of model collapse (Zhu et al.,\n2024). Mathematical and programming corpora still remain core components of our training\ndata, and we further enhance the coding capabilities of DeepSeek-V4 series by incorporating\nagentic data during the mid-training phase. For multilingual data, we build a larger corpus\nfor DeepSeek-V4, improving its capture of long-tail knowledge across different cultures. For\nDeepSeek-V4, we place a particular emphasis on long-document data curation, prioritizing\nscientific papers, technical reports, and other materials that reflect unique academic values.\nCombining all the above, our pre-training corpus comprises more than 32T tokens, containing\nmathematical contents, codes, web pages, long documents, and other high-quality categories.\nFor pre-training data, we largely follow the same pre-processing strategies of DeepSeek-\nV3. For tokenization, on top of the DeepSeek-V3 tokenizer, we introduce a few special tokens\nfor context construction, and still remain the vocabulary size to be 128K. We also inherit the\ntoken-splitting (DeepSeek-AI, 2024) and Fill-in-Middle (FIM) (DeepSeek-AI, 2024) strategies\nfrom DeepSeek-V3. Inspired by Ding et al. (2024), we pack documents from different sources\ninto appropriate sequences to minimize sample truncation. Different from DeepSeek-V3, we\nemploy sample-level attention masking during pre-training.\n4.2. Pre-Training Setups\n4.2.1. Model Setups\nDeepSeek-V4-Flash.We set the number of Transformer layers to 43 and the hidden dimension\n\ud835\udc51to 4096. For the first two layers, we use pure sliding window attention. For the subsequent\nlayers, CSA and HCA are used in an interleaved manner. For CSA, we set the compression rate\n\ud835\udc5ato 4, the number of indexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and\nthe number of KV entries selected for sparse attention (i.e., attention top-k) to 512. For HCA,\nwe set the compression rate \ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query\nheads\ud835\udc5b\u210eto 64, the head dimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1024.\nThe number of output projection groups \ud835\udc54is set to 8, and the dimension of each intermediate\nattention output \ud835\udc51\ud835\udc54is set to 1024. For the additional branch of sliding window attention, the\nwindow size \ud835\udc5bwinis set to 128. We employ MoE layers in all Transformer blocks, but use the\nHash routing strategy for the first 3 MoE layers. Each MoE layer consists of 1 shared expert and\n256 routed experts, where the intermediate hidden dimension of each expert is 2048. Among the\nrouted experts, 6 experts will be activated for each token. The multi-token prediction depth is\nset to 1. As formHC, the expansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp\niterations \ud835\udc61maxis set to 20. Under this configuration, DeepSeek-V4-Flash comprises 284B total\nparameters, of which 13B are activated for each token.\n24", | |
| "nodes": [ | |
| { | |
| "title": "Model Setups", | |
| "node_id": "0026", | |
| "start_index": 24, | |
| "end_index": 25, | |
| "summary": "The partial document primarily covers the pre-training phase of the DeepSeek-V4 series, detailing data construction and training configurations. It discusses the creation of a diverse, high-quality training corpus exceeding 32 trillion tokens, emphasizing long-context data, multilingual content, and specialized domains like mathematics, programming, and scientific literature. The document outlines enhancements over the previous DeepSeek-V3, including refined tokenization, attention mechanisms, and data packing strategies. It then describes the architectural setups for two model variants, DeepSeek-V4-Flash and DeepSeek-V4-Pro, specifying their Transformer layer counts, hidden dimensions, attention types (sliding window, CSA, HCA), MoE layer configurations, and parameter scales. Finally, it details the training setups for both models, including optimizer choices (Muon and AdamW), hyperparameters, batch size scheduling, learning rate schedules, sequence length progression, and the introduction of sparse attention mechanisms during training.", | |
| "text": "Depending on specific deployment scenarios, we select the most suitable strategy to achieve the\ndesired trade-off between storage and computation.\n4. Pre-Training\n4.1. Data Construction\nOn top of the pre-training data of DeepSeek-V3, we endeavor to construct a more diverse and\nhigher-quality training corpus with longer effective contexts. We continually refine our data con-\nstruction pipelines. For web-sourced data, we implement filtering strategies to remove batched\nauto-generated and templated content, thereby mitigating the risk of model collapse (Zhu et al.,\n2024). Mathematical and programming corpora still remain core components of our training\ndata, and we further enhance the coding capabilities of DeepSeek-V4 series by incorporating\nagentic data during the mid-training phase. For multilingual data, we build a larger corpus\nfor DeepSeek-V4, improving its capture of long-tail knowledge across different cultures. For\nDeepSeek-V4, we place a particular emphasis on long-document data curation, prioritizing\nscientific papers, technical reports, and other materials that reflect unique academic values.\nCombining all the above, our pre-training corpus comprises more than 32T tokens, containing\nmathematical contents, codes, web pages, long documents, and other high-quality categories.\nFor pre-training data, we largely follow the same pre-processing strategies of DeepSeek-\nV3. For tokenization, on top of the DeepSeek-V3 tokenizer, we introduce a few special tokens\nfor context construction, and still remain the vocabulary size to be 128K. We also inherit the\ntoken-splitting (DeepSeek-AI, 2024) and Fill-in-Middle (FIM) (DeepSeek-AI, 2024) strategies\nfrom DeepSeek-V3. Inspired by Ding et al. (2024), we pack documents from different sources\ninto appropriate sequences to minimize sample truncation. Different from DeepSeek-V3, we\nemploy sample-level attention masking during pre-training.\n4.2. Pre-Training Setups\n4.2.1. Model Setups\nDeepSeek-V4-Flash.We set the number of Transformer layers to 43 and the hidden dimension\n\ud835\udc51to 4096. For the first two layers, we use pure sliding window attention. For the subsequent\nlayers, CSA and HCA are used in an interleaved manner. For CSA, we set the compression rate\n\ud835\udc5ato 4, the number of indexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and\nthe number of KV entries selected for sparse attention (i.e., attention top-k) to 512. For HCA,\nwe set the compression rate \ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query\nheads\ud835\udc5b\u210eto 64, the head dimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1024.\nThe number of output projection groups \ud835\udc54is set to 8, and the dimension of each intermediate\nattention output \ud835\udc51\ud835\udc54is set to 1024. For the additional branch of sliding window attention, the\nwindow size \ud835\udc5bwinis set to 128. We employ MoE layers in all Transformer blocks, but use the\nHash routing strategy for the first 3 MoE layers. Each MoE layer consists of 1 shared expert and\n256 routed experts, where the intermediate hidden dimension of each expert is 2048. Among the\nrouted experts, 6 experts will be activated for each token. The multi-token prediction depth is\nset to 1. As formHC, the expansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp\niterations \ud835\udc61maxis set to 20. Under this configuration, DeepSeek-V4-Flash comprises 284B total\nparameters, of which 13B are activated for each token.\n24DeepSeek-V4-Pro.We set the number of Transformer layers to 61 and the hidden dimension\n\ud835\udc51to 7168. For the first two layers, we use HCA. For the subsequent layers, CSA and HCA are\nused in an interleaved manner. For CSA, we set the compression rate \ud835\udc5ato 4, the number of\nindexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and the number of KV entries\nselected for sparse attention (i.e., attention top-k) to 1024. For HCA, we set the compression\nrate\ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query heads \ud835\udc5b\u210eto 128, the head\ndimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1536. The number of output\nprojection groups \ud835\udc54is set to 16, and the dimension of each intermediate attention output \ud835\udc51\ud835\udc54is set\nto 1024. For the additional branch of sliding window attention, the window size \ud835\udc5bwinis set to\n128. We employ MoE layers in all Transformer blocks, but use the Hash routing strategy for the\nfirst 3 MoE layers. Each MoE layer consists of 1 shared expert and 384 routed experts, where\nthe intermediate hidden dimension of each expert is 3072. Among the routed experts, 6 experts\nwill be activated for each token. The multi-token prediction depth is set to 1. As formHC, the\nexpansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp iterations \ud835\udc61maxis set to 20.\nUnder this configuration, DeepSeek-V4-Pro comprises 1.6T total parameters, of which 49B are\nactivated for each token.\n4.2.2. Training Setups\nDeepSeek-V4-Flash.We employ the Muon optimizer (Jordan et al., 2024; Liu et al., 2025) for\nthe majority of parameters, but use the AdamW optimizer (Loshchilov and Hutter, 2017) for the\nembedding module, the prediction head module, and the weights of all RMSNorm modules. For\nAdamW, we set its hyper-parameters to \ud835\udefd1=0.9,\ud835\udefd2=0.95,\ud835\udf00=10\u221220, and weight_decay= 0.1.\nFor Muon, we set the momentum to 0.95 and the weight decay to 0.1, and rescale the RMS of each\nupdate matrix to 0.18 for reutilization of the AdamW learning rate. We train DeepSeek-V4-Flash\non 32T tokens, and as in DeepSeek-V3, we also employ a batch size scheduling strategy that\nincreases the batch size (in tokens) from a small size to 75.5M and then keeps it at 75.5M during\nmost of the training. The learning rate is linearly warmed up in the first 2000 steps, maintained\nat 2.7\u00d710\u22124for most of the training. Near the end of the training, we finally decay the learning\nrate to 2.7\u00d710\u22125following a cosine schedule. The training starts with a sequence length of 4K,\nand we gradually extend the training sequence length to 16K, 64K, and 1M. As for the setups of\nsparse attention, we first warmup the model with dense attention for the first 1T tokens, and\nintroduce sparse attention at the sequence length of 64K and keep sparse attention during the\nrest of the training. When introducing attention sparsity, we first set a short stage to warm up\nthe lightning indexer in CSA, and then train the model with sparse attention for most of the\ntraining. For auxiliary-loss-free load balancing, we set the bias update speed to 0.001. For the\nbalance loss, we set its loss weight to 0.0001 to avoid extreme imbalance within single sequences.\nThe MTP loss weight is set to 0.3 for most of the training, and to 0.1 upon the start of learning\nrate decay.\nDeepSeek-V4-Pro.Except for specific values of hyper-parameters, the training setup of\nDeepSeek-V4-Pro is largely consistent with that of DeepSeek-V4-Flash. We employ the Muon op-\ntimizer for the majority of parameters, but use the AdamW optimizer for the embedding module,\nthe prediction head module, and the weights of all RMSNorm modules. The hyper-parameters\nof AdamW and Muon are the same as those of DeepSeek-V4-Flash. We train DeepSeek-V4-Pro\non 33T tokens, and also employ a batch size scheduling strategy, with the maximum batch\nsize being 94.4M tokens. The learning rate scheduling strategy is largely the same as that of\nDeepSeek-V4-Flash, but the peak learning rate is set to 2.0 \u00d710\u22124and the end learning rate is set\nto 2.0\u00d710\u22125. The training also starts with a sequence length of 4K, and the length is gradually\n25" | |
| }, | |
| { | |
| "title": "Training Setups", | |
| "node_id": "0027", | |
| "start_index": 25, | |
| "end_index": 26, | |
| "summary": "The partial document details the architectural design, training configurations, and stability techniques for the DeepSeek-V4 series of large-scale Transformer models, specifically DeepSeek-V4-Pro and DeepSeek-V4-Flash. It covers the model architecture, including the number of Transformer layers, hidden dimensions, and the use of hybrid attention mechanisms (HCA and CSA) with specific hyperparameters such as compression rates, query heads, and sparse attention settings. The document also describes the incorporation of Mixture of Experts (MoE) layers with Hash routing and expert activation details.\n\nIn the training setups section, it outlines the use of two optimizers\u2014Muon for most parameters and AdamW for specific modules\u2014along with their hyperparameters, batch size scheduling, learning rate schedules, and progressive sequence length increases. Differences between DeepSeek-V4-Pro and DeepSeek-V4-Flash in training token counts, batch sizes, and learning rates are highlighted, as well as the staged introduction of sparse attention.\n\nFinally, the document addresses the significant training instability challenges encountered with trillion-parameter MoE models. It introduces two practical solutions to mitigate these issues: Anticipatory Routing, which decouples routing index computation from current network parameters to prevent loss spikes with minimal overhead, and SwiGLU Clamping, which constrains activation ranges to suppress outliers and stabilize training without performance loss. These techniques are presented as empirical findings aimed at improving training robustness.", | |
| "text": "DeepSeek-V4-Pro.We set the number of Transformer layers to 61 and the hidden dimension\n\ud835\udc51to 7168. For the first two layers, we use HCA. For the subsequent layers, CSA and HCA are\nused in an interleaved manner. For CSA, we set the compression rate \ud835\udc5ato 4, the number of\nindexer query heads \ud835\udc5b\ud835\udc3c\n\u210eto 64, the indexer head dimension \ud835\udc50\ud835\udc3cto 128, and the number of KV entries\nselected for sparse attention (i.e., attention top-k) to 1024. For HCA, we set the compression\nrate\ud835\udc5a\u2032to 128. For both CSA and HCA, we set the number of query heads \ud835\udc5b\u210eto 128, the head\ndimension \ud835\udc50to 512, and the query compression dimension \ud835\udc51\ud835\udc50to 1536. The number of output\nprojection groups \ud835\udc54is set to 16, and the dimension of each intermediate attention output \ud835\udc51\ud835\udc54is set\nto 1024. For the additional branch of sliding window attention, the window size \ud835\udc5bwinis set to\n128. We employ MoE layers in all Transformer blocks, but use the Hash routing strategy for the\nfirst 3 MoE layers. Each MoE layer consists of 1 shared expert and 384 routed experts, where\nthe intermediate hidden dimension of each expert is 3072. Among the routed experts, 6 experts\nwill be activated for each token. The multi-token prediction depth is set to 1. As formHC, the\nexpansion factor \ud835\udc5bhcis set to 4, and the number of Sinkhorn-Knopp iterations \ud835\udc61maxis set to 20.\nUnder this configuration, DeepSeek-V4-Pro comprises 1.6T total parameters, of which 49B are\nactivated for each token.\n4.2.2. Training Setups\nDeepSeek-V4-Flash.We employ the Muon optimizer (Jordan et al., 2024; Liu et al., 2025) for\nthe majority of parameters, but use the AdamW optimizer (Loshchilov and Hutter, 2017) for the\nembedding module, the prediction head module, and the weights of all RMSNorm modules. For\nAdamW, we set its hyper-parameters to \ud835\udefd1=0.9,\ud835\udefd2=0.95,\ud835\udf00=10\u221220, and weight_decay= 0.1.\nFor Muon, we set the momentum to 0.95 and the weight decay to 0.1, and rescale the RMS of each\nupdate matrix to 0.18 for reutilization of the AdamW learning rate. We train DeepSeek-V4-Flash\non 32T tokens, and as in DeepSeek-V3, we also employ a batch size scheduling strategy that\nincreases the batch size (in tokens) from a small size to 75.5M and then keeps it at 75.5M during\nmost of the training. The learning rate is linearly warmed up in the first 2000 steps, maintained\nat 2.7\u00d710\u22124for most of the training. Near the end of the training, we finally decay the learning\nrate to 2.7\u00d710\u22125following a cosine schedule. The training starts with a sequence length of 4K,\nand we gradually extend the training sequence length to 16K, 64K, and 1M. As for the setups of\nsparse attention, we first warmup the model with dense attention for the first 1T tokens, and\nintroduce sparse attention at the sequence length of 64K and keep sparse attention during the\nrest of the training. When introducing attention sparsity, we first set a short stage to warm up\nthe lightning indexer in CSA, and then train the model with sparse attention for most of the\ntraining. For auxiliary-loss-free load balancing, we set the bias update speed to 0.001. For the\nbalance loss, we set its loss weight to 0.0001 to avoid extreme imbalance within single sequences.\nThe MTP loss weight is set to 0.3 for most of the training, and to 0.1 upon the start of learning\nrate decay.\nDeepSeek-V4-Pro.Except for specific values of hyper-parameters, the training setup of\nDeepSeek-V4-Pro is largely consistent with that of DeepSeek-V4-Flash. We employ the Muon op-\ntimizer for the majority of parameters, but use the AdamW optimizer for the embedding module,\nthe prediction head module, and the weights of all RMSNorm modules. The hyper-parameters\nof AdamW and Muon are the same as those of DeepSeek-V4-Flash. We train DeepSeek-V4-Pro\non 33T tokens, and also employ a batch size scheduling strategy, with the maximum batch\nsize being 94.4M tokens. The learning rate scheduling strategy is largely the same as that of\nDeepSeek-V4-Flash, but the peak learning rate is set to 2.0 \u00d710\u22124and the end learning rate is set\nto 2.0\u00d710\u22125. The training also starts with a sequence length of 4K, and the length is gradually\n25extended to 16K, 64K, and 1M. Compared with DeepSeek-V4-Flash, DeepSeek-V4-Pro starts\nwith a longer stage of dense attention, and the strategy of introducing sparse attention is the\nsame as DeepSeek-V4-Flash, following a two-stage training method. For auxiliary-loss-free load\nbalancing, we set the bias update speed to 0.001. For the balance loss, we set its loss weight to\n0.0001 to avoid extreme imbalance within single sequences. The MTP loss weight is set to 0.3 for\nmost of the training, and to 0.1 upon the start of learning rate decay.\n4.2.3. Mitigating Training Instability\nTraining trillion-parameter MoE models presents significant stability challenges, and DeepSeek-\nV4 series are no exception. We encountered notable instability challenges during training.\nWhile simple rollbacks could temporarily restore the training state, they proved inadequate as a\nlong-term solution because they do not prevent the recurrence of loss spikes. Empirically, we\nidentified that the occurrence of spikes is consistently tied to outliers in the MoE layers, and the\nrouting mechanism itself appears to exacerbate the emergence of these outliers. Therefore, we\nsought to tackle this issue from two dimensions: breaking the vicious cycle induced by routing,\nand directly suppressing anomalous values. Fortunately, we discovered two practical techniques\nthat effectively maintain training stability. Although a comprehensive theoretical understanding\nof their underlying mechanisms remains an open question for now, we are sharing them openly\nto foster further exploration by the community.\nAnticipatory Routing.We found that decoupling the synchronous updates of the backbone\nnetwork and the routing network significantly improves training stability. Consequently, at step\n\ud835\udc61, we use the current network parameters \ud835\udf03\ud835\udc61for feature computation, but the routing indices are\ncomputed and applied using the historical network parameters \ud835\udf03\ud835\udc61\u2212\u0394\ud835\udc61. In practice, to circumvent\nthe overhead of loading model parameters twice, we fetch the data for step \ud835\udc61in advance at\nstep\ud835\udc61\u2212\u0394\ud835\udc61 . We \"anticipatorily\" compute and cache the routing indices to be used later at step\n\ud835\udc61, which is why we name this approach Anticipatory Routing. We also heavily optimized this\nat the infrastructure level. First, given that pre-computing the routing indices only requires\na single forward pass over the data, we carefully orchestrated the pipeline execution and the\noverlapping of computation with Expert Parallelism (EP) communication, successfully bounding\nthe additional wall-clock time overhead of Anticipatory Routing to approximately 20%. Second,\nwe introduced an automatic detection mechanism that triggers a short rollback and activates\nAnticipatory Routing exclusively when a loss spike occurs; after operating in this mode for a\ncertain period, the system reverts to standard training. Ultimately, this dynamic application\nallows us to avert loss spikes with negligible overall additional training overhead, all without\ncompromising model performance.\nSwiGLU Clamping.In previous literature (Bello et al., 2017; Riviere et al., 2024), clamping\nhas been explicitly utilized to constrain numerical ranges, thereby enhancing training stability.\nIn our actual training runs, we empirically found that applying SwiGLU clamping (OpenAI,\n2025) effectively eliminates outliers and substantially aids in stabilizing the training process,\nwithout compromising performance. Throughout the training of both DeepSeek-V4-Flash and\nDeepSeek-V4-Pro, we clamped the linear component of SwiGLU to the range of [\u221210, 10], while\ncapping the upper bound of the gate component at 10.\n26" | |
| }, | |
| { | |
| "title": "Mitigating Training Instability", | |
| "node_id": "0028", | |
| "start_index": 26, | |
| "end_index": 26, | |
| "summary": "The partial document discusses advanced training techniques and stability improvements for large-scale Mixture of Experts (MoE) models, specifically the DeepSeek-V4 series. It covers the model architecture differences between DeepSeek-V4-Flash and DeepSeek-V4-Pro, including attention mechanisms and training loss weight settings. A significant focus is on addressing training instability caused by outliers in MoE layers and routing mechanisms. Two key methods are introduced: Anticipatory Routing, which decouples routing updates from backbone network updates to prevent loss spikes with minimal overhead, and SwiGLU Clamping, which constrains numerical ranges in activation functions to suppress outliers and stabilize training. The document emphasizes practical solutions to maintain training stability in trillion-parameter models while acknowledging that theoretical understanding remains incomplete.", | |
| "text": "extended to 16K, 64K, and 1M. Compared with DeepSeek-V4-Flash, DeepSeek-V4-Pro starts\nwith a longer stage of dense attention, and the strategy of introducing sparse attention is the\nsame as DeepSeek-V4-Flash, following a two-stage training method. For auxiliary-loss-free load\nbalancing, we set the bias update speed to 0.001. For the balance loss, we set its loss weight to\n0.0001 to avoid extreme imbalance within single sequences. The MTP loss weight is set to 0.3 for\nmost of the training, and to 0.1 upon the start of learning rate decay.\n4.2.3. Mitigating Training Instability\nTraining trillion-parameter MoE models presents significant stability challenges, and DeepSeek-\nV4 series are no exception. We encountered notable instability challenges during training.\nWhile simple rollbacks could temporarily restore the training state, they proved inadequate as a\nlong-term solution because they do not prevent the recurrence of loss spikes. Empirically, we\nidentified that the occurrence of spikes is consistently tied to outliers in the MoE layers, and the\nrouting mechanism itself appears to exacerbate the emergence of these outliers. Therefore, we\nsought to tackle this issue from two dimensions: breaking the vicious cycle induced by routing,\nand directly suppressing anomalous values. Fortunately, we discovered two practical techniques\nthat effectively maintain training stability. Although a comprehensive theoretical understanding\nof their underlying mechanisms remains an open question for now, we are sharing them openly\nto foster further exploration by the community.\nAnticipatory Routing.We found that decoupling the synchronous updates of the backbone\nnetwork and the routing network significantly improves training stability. Consequently, at step\n\ud835\udc61, we use the current network parameters \ud835\udf03\ud835\udc61for feature computation, but the routing indices are\ncomputed and applied using the historical network parameters \ud835\udf03\ud835\udc61\u2212\u0394\ud835\udc61. In practice, to circumvent\nthe overhead of loading model parameters twice, we fetch the data for step \ud835\udc61in advance at\nstep\ud835\udc61\u2212\u0394\ud835\udc61 . We \"anticipatorily\" compute and cache the routing indices to be used later at step\n\ud835\udc61, which is why we name this approach Anticipatory Routing. We also heavily optimized this\nat the infrastructure level. First, given that pre-computing the routing indices only requires\na single forward pass over the data, we carefully orchestrated the pipeline execution and the\noverlapping of computation with Expert Parallelism (EP) communication, successfully bounding\nthe additional wall-clock time overhead of Anticipatory Routing to approximately 20%. Second,\nwe introduced an automatic detection mechanism that triggers a short rollback and activates\nAnticipatory Routing exclusively when a loss spike occurs; after operating in this mode for a\ncertain period, the system reverts to standard training. Ultimately, this dynamic application\nallows us to avert loss spikes with negligible overall additional training overhead, all without\ncompromising model performance.\nSwiGLU Clamping.In previous literature (Bello et al., 2017; Riviere et al., 2024), clamping\nhas been explicitly utilized to constrain numerical ranges, thereby enhancing training stability.\nIn our actual training runs, we empirically found that applying SwiGLU clamping (OpenAI,\n2025) effectively eliminates outliers and substantially aids in stabilizing the training process,\nwithout compromising performance. Throughout the training of both DeepSeek-V4-Flash and\nDeepSeek-V4-Pro, we clamped the linear component of SwiGLU to the range of [\u221210, 10], while\ncapping the upper bound of the gate component at 10.\n26" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Evaluations", | |
| "node_id": "0029", | |
| "start_index": 27, | |
| "end_index": 27, | |
| "summary": "The partial document section focuses on the evaluation of base models within the DeepSeek series, detailing the benchmarks used and the comparative results. It outlines four key evaluation dimensions: world knowledge, language understanding and reasoning, coding and mathematics, and long-context processing, listing specific benchmark datasets for each category. The evaluation results compare three DeepSeek base models\u2014V3.2, V4-Flash, and V4-Pro\u2014under consistent testing conditions. The findings highlight that DeepSeek-V4-Flash-Base, despite having fewer parameters, outperforms DeepSeek-V3.2-Base, particularly in world knowledge and long-context tasks, due to architectural and training improvements. DeepSeek-V4-Pro-Base shows a significant performance leap over both predecessors, achieving superior results across nearly all benchmarks, especially in knowledge-intensive, reasoning, coding, and long-context evaluations, establishing it as the most capable model in the series.", | |
| "text": "4.3. Evaluations\n4.3.1. Evaluation Benchmarks\nFor the evaluation of the base models, we consider benchmarks spanning four key dimensions:\nworld knowledge, language understanding and reasoning, coding and mathematics, and long-\ncontext processing.\nWorld knowledgebenchmarks include AGIEval (Zhong et al., 2023), C-Eval (Huang et al.,\n2023), CMMLU (Li et al., 2023) MMLU (Hendrycks et al., 2020), MMLU-Redux (Gema et al.,\n2024), MMLU-Pro (Wang et al., 2024b), MMMLU (OpenAI, 2024a), MultiLoKo (Hupkes and\nBogoychev, 2025), Simple-QA verified (Haas et al., 2025), SuperGPQA (Du et al., 2025), FACTS\nParametric (Cheng et al., 2025), and TriviaQA (Joshi et al., 2017).\nLanguage understanding and reasoningbenchmarks include BigBench Hard (BBH) (Suzgun\net al., 2022), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), CLUEWSC (Xu et al., 2020),\nand WinoGrande (Sakaguchi et al., 2019).\nCoding and mathematicalbenchmarks include BigCodeBench (Zhuo et al., 2025), Hu-\nmanEval (Chen et al., 2021), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MGSM\n(Shi et al., 2023), and CMath (Wei et al., 2023).\nLong contextbenchmarks include LongBench-V2 (Bai et al., 2025b).\n4.3.2. Evaluation Results\nIn Table 1, we provide a detailed comparison of the base models for DeepSeek-V3.2, DeepSeek-\nV4-Flash, and DeepSeek-V4-Pro, all evaluated under a unified internal framework with strictly\nconsistent settings.\nComparing DeepSeek-V4-Flash-Base with DeepSeek-V3.2-Base reveals a compelling ef-\nficiency story. Despite utilizing a substantially smaller number of both activated and total\nparameters, DeepSeek-V4-Flash-Base outperforms DeepSeek-V3.2-Base across a wide array of\nbenchmarks. This advantage is especially evident in world knowledge tasks and challenging\nlong-context scenarios. These results underscore that architectural improvements, refined data\nquality, and training optimizations in DeepSeek-V4-Flash-Base yield superior performance even\nwith a more compact parameter budget, effectively surpassing the larger DeepSeek-V3.2-Base\non the majority of evaluations.\nFurthermore, DeepSeek-V4-Pro-Base demonstrates a further, decisive leap in capability,\nestablishing near-universal dominance over both DeepSeek-V3.2-Base and DeepSeek-V4-Flash-\nBase. With improvements across almost all categories, DeepSeek-V4-Pro-Base reaches new\nperformance highs among DeepSeek base models on the most demanding benchmarks. On\nknowledge-intensive evaluations, it delivers dramatic gains, while also substantially advancing\nlong-context understanding. On most reasoning and code benchmarks, DeepSeek-V4-Pro-Base\nalso exceeds both previous models. This comprehensive uplift confirms DeepSeek-V4-Pro-Base\nas the strongest foundation model in the DeepSeek series, outperforming its predecessors across\nthe spectrum of knowledge, reasoning, coding, and long-context capabilities.\n27", | |
| "nodes": [ | |
| { | |
| "title": "Evaluation Benchmarks", | |
| "node_id": "0030", | |
| "start_index": 27, | |
| "end_index": 27, | |
| "summary": "The partial document section focuses on the evaluation of base models within the DeepSeek series, detailing the benchmarks used and the comparative results. It outlines four key evaluation dimensions: world knowledge, language understanding and reasoning, coding and mathematics, and long-context processing, listing specific benchmark datasets for each category. The evaluation results compare three DeepSeek base models\u2014DeepSeek-V3.2, DeepSeek-V4-Flash, and DeepSeek-V4-Pro\u2014under consistent testing conditions. Findings highlight that DeepSeek-V4-Flash-Base, despite having fewer parameters, outperforms DeepSeek-V3.2-Base, particularly in world knowledge and long-context tasks, due to architectural and training improvements. DeepSeek-V4-Pro-Base shows a significant overall performance leap, surpassing both earlier models across nearly all benchmarks, establishing it as the most capable foundation model in the series for knowledge, reasoning, coding, and long-context understanding.", | |
| "text": "4.3. Evaluations\n4.3.1. Evaluation Benchmarks\nFor the evaluation of the base models, we consider benchmarks spanning four key dimensions:\nworld knowledge, language understanding and reasoning, coding and mathematics, and long-\ncontext processing.\nWorld knowledgebenchmarks include AGIEval (Zhong et al., 2023), C-Eval (Huang et al.,\n2023), CMMLU (Li et al., 2023) MMLU (Hendrycks et al., 2020), MMLU-Redux (Gema et al.,\n2024), MMLU-Pro (Wang et al., 2024b), MMMLU (OpenAI, 2024a), MultiLoKo (Hupkes and\nBogoychev, 2025), Simple-QA verified (Haas et al., 2025), SuperGPQA (Du et al., 2025), FACTS\nParametric (Cheng et al., 2025), and TriviaQA (Joshi et al., 2017).\nLanguage understanding and reasoningbenchmarks include BigBench Hard (BBH) (Suzgun\net al., 2022), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), CLUEWSC (Xu et al., 2020),\nand WinoGrande (Sakaguchi et al., 2019).\nCoding and mathematicalbenchmarks include BigCodeBench (Zhuo et al., 2025), Hu-\nmanEval (Chen et al., 2021), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MGSM\n(Shi et al., 2023), and CMath (Wei et al., 2023).\nLong contextbenchmarks include LongBench-V2 (Bai et al., 2025b).\n4.3.2. Evaluation Results\nIn Table 1, we provide a detailed comparison of the base models for DeepSeek-V3.2, DeepSeek-\nV4-Flash, and DeepSeek-V4-Pro, all evaluated under a unified internal framework with strictly\nconsistent settings.\nComparing DeepSeek-V4-Flash-Base with DeepSeek-V3.2-Base reveals a compelling ef-\nficiency story. Despite utilizing a substantially smaller number of both activated and total\nparameters, DeepSeek-V4-Flash-Base outperforms DeepSeek-V3.2-Base across a wide array of\nbenchmarks. This advantage is especially evident in world knowledge tasks and challenging\nlong-context scenarios. These results underscore that architectural improvements, refined data\nquality, and training optimizations in DeepSeek-V4-Flash-Base yield superior performance even\nwith a more compact parameter budget, effectively surpassing the larger DeepSeek-V3.2-Base\non the majority of evaluations.\nFurthermore, DeepSeek-V4-Pro-Base demonstrates a further, decisive leap in capability,\nestablishing near-universal dominance over both DeepSeek-V3.2-Base and DeepSeek-V4-Flash-\nBase. With improvements across almost all categories, DeepSeek-V4-Pro-Base reaches new\nperformance highs among DeepSeek base models on the most demanding benchmarks. On\nknowledge-intensive evaluations, it delivers dramatic gains, while also substantially advancing\nlong-context understanding. On most reasoning and code benchmarks, DeepSeek-V4-Pro-Base\nalso exceeds both previous models. This comprehensive uplift confirms DeepSeek-V4-Pro-Base\nas the strongest foundation model in the DeepSeek series, outperforming its predecessors across\nthe spectrum of knowledge, reasoning, coding, and long-context capabilities.\n27" | |
| }, | |
| { | |
| "title": "Evaluation Results", | |
| "node_id": "0031", | |
| "start_index": 27, | |
| "end_index": 28, | |
| "summary": "The partial document covers the evaluation methodology and results for the DeepSeek series of base models, focusing on three versions: DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-Pro-Base. It details the evaluation benchmarks used, which span four key areas: world knowledge, language understanding and reasoning, coding and mathematics, and long-context processing. The document presents a comparative analysis showing that DeepSeek-V4-Flash-Base achieves better performance than DeepSeek-V3.2-Base despite having fewer parameters, highlighting improvements in architecture, data quality, and training. Furthermore, DeepSeek-V4-Pro-Base demonstrates a significant performance leap, outperforming both predecessors across nearly all benchmark categories, especially in knowledge-intensive tasks and long-context understanding. The document also includes a detailed results table comparing model scores across benchmarks. Additionally, it introduces the post-training phase for the DeepSeek-V4 series, noting a key methodological change where mixed Reinforcement Learning was replaced by On-Policy Distillation, and describes the specialist training process involving fine-tuning and Reinforcement Learning with domain-specific prompts using the Group Relative Policy Optimization algorithm.", | |
| "text": "4.3. Evaluations\n4.3.1. Evaluation Benchmarks\nFor the evaluation of the base models, we consider benchmarks spanning four key dimensions:\nworld knowledge, language understanding and reasoning, coding and mathematics, and long-\ncontext processing.\nWorld knowledgebenchmarks include AGIEval (Zhong et al., 2023), C-Eval (Huang et al.,\n2023), CMMLU (Li et al., 2023) MMLU (Hendrycks et al., 2020), MMLU-Redux (Gema et al.,\n2024), MMLU-Pro (Wang et al., 2024b), MMMLU (OpenAI, 2024a), MultiLoKo (Hupkes and\nBogoychev, 2025), Simple-QA verified (Haas et al., 2025), SuperGPQA (Du et al., 2025), FACTS\nParametric (Cheng et al., 2025), and TriviaQA (Joshi et al., 2017).\nLanguage understanding and reasoningbenchmarks include BigBench Hard (BBH) (Suzgun\net al., 2022), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), CLUEWSC (Xu et al., 2020),\nand WinoGrande (Sakaguchi et al., 2019).\nCoding and mathematicalbenchmarks include BigCodeBench (Zhuo et al., 2025), Hu-\nmanEval (Chen et al., 2021), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MGSM\n(Shi et al., 2023), and CMath (Wei et al., 2023).\nLong contextbenchmarks include LongBench-V2 (Bai et al., 2025b).\n4.3.2. Evaluation Results\nIn Table 1, we provide a detailed comparison of the base models for DeepSeek-V3.2, DeepSeek-\nV4-Flash, and DeepSeek-V4-Pro, all evaluated under a unified internal framework with strictly\nconsistent settings.\nComparing DeepSeek-V4-Flash-Base with DeepSeek-V3.2-Base reveals a compelling ef-\nficiency story. Despite utilizing a substantially smaller number of both activated and total\nparameters, DeepSeek-V4-Flash-Base outperforms DeepSeek-V3.2-Base across a wide array of\nbenchmarks. This advantage is especially evident in world knowledge tasks and challenging\nlong-context scenarios. These results underscore that architectural improvements, refined data\nquality, and training optimizations in DeepSeek-V4-Flash-Base yield superior performance even\nwith a more compact parameter budget, effectively surpassing the larger DeepSeek-V3.2-Base\non the majority of evaluations.\nFurthermore, DeepSeek-V4-Pro-Base demonstrates a further, decisive leap in capability,\nestablishing near-universal dominance over both DeepSeek-V3.2-Base and DeepSeek-V4-Flash-\nBase. With improvements across almost all categories, DeepSeek-V4-Pro-Base reaches new\nperformance highs among DeepSeek base models on the most demanding benchmarks. On\nknowledge-intensive evaluations, it delivers dramatic gains, while also substantially advancing\nlong-context understanding. On most reasoning and code benchmarks, DeepSeek-V4-Pro-Base\nalso exceeds both previous models. This comprehensive uplift confirms DeepSeek-V4-Pro-Base\nas the strongest foundation model in the DeepSeek series, outperforming its predecessors across\nthe spectrum of knowledge, reasoning, coding, and long-context capabilities.\n27Table 1|Comparison among DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-\nPro-Base. All models are evaluated in our internal framework and share the same evaluation\nsetting. Scores with a gap not exceeding 0.3 are considered to be at the same level. The highest\nscore in each row is inbold font, and the second is underlined .\nBenchmark(Metric)# ShotsDeepSeek-V3.2 DeepSeek-V4-Flash DeepSeek-V4-Pro\nBase Base Base\nArchitecture - MoE MoE MoE\n# Activated Params - 37B 13B 49B\n# Total Params - 671B 284B 1.6T\nWorld Knowl.AGIEval(EM)0-shot 80.1 82.6 83.1\nMMLU(EM)5-shot 87.8 88.7 90.1\nMMLU-Redux(EM)5-shot 87.5 89.4 90.8\nMMLU-Pro(EM)5-shot 65.5 68.3 73.5\nMMMLU(EM)5-shot 87.9 88.8 90.3\nC-Eval(EM)5-shot 90.4 92.1 93.1\nCMMLU(EM)5-shot 88.9 90.4 90.8\nMultiLoKo(EM)5-shot 38.7 42.2 51.1\nSimple-QA verified(EM)25-shot 28.3 30.1 55.2\nSuperGPQA(EM)5-shot 45.0 46.5 53.9\nFACTS Parametric(EM)25-shot 27.1 33.9 62.6\nTriviaQA(EM)5-shot 83.3 82.885.6\nLang. & Reas.BBH(EM)3-shot 87.686.9 87.5\nDROP(F1)1-shot 88.2 88.6 88.7\nHellaSwag(EM)0-shot 86.4 85.788.0\nWinoGrande(EM)0-shot 78.9 79.5 81.5\nCLUEWSC(EM)5-shot 83.5 82.285.2\nCode & MathBigCodeBench(Pass@1)3-shot 63.956.8 59.2\nHumanEval(Pass@1)0-shot 62.8 69.5 76.8\nGSM8K(EM)8-shot 91.1 90.892.6\nMATH(EM)4-shot 60.5 57.464.5\nMGSM(EM)8-shot 81.385.784.4\nCMath(EM)3-shot 92.6 93.690.9\nLong Context LongBench-V2(EM)1-shot 40.2 44.7 51.5\n5. Post-Training\n5.1. Post-Training Pipeline\nFollowing pre-training, we conducted a post-training phase to yield the final models of DeepSeek-\nV4 series. Although the training pipeline largely mirrored that of DeepSeek-V3.2, a critical\nmethodological substitution was made: the mixed Reinforcement Learning (RL) stage was\nentirely replaced by On-Policy Distillation (OPD; Gu et al., 2024; Lu and Lab, 2025).\n5.1.1. Specialist Training\nThe development of domain specialists was conducted by adapting the DeepSeek-V3.2 training\npipeline. Specifically, each model was sequentially optimized through an initial fine-tuning\nphase and subsequent Reinforcement Learning (RL) guided by domain-specific prompts and re-\nward signals. For the RL stage, we implemented the Group Relative Policy Optimization (GRPO)\nalgorithm, maintaining hyper-parameters closely aligned with our prior research (DeepSeek-AI,\n2025; DeepSeek-AI, 2025).\n28" | |
| } | |
| ] | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Post-Training", | |
| "node_id": "0032", | |
| "start_index": 28, | |
| "end_index": 28, | |
| "summary": "The partial document presents a comparative analysis of three DeepSeek model versions\u2014DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-Pro-Base\u2014evaluated under consistent internal settings across various benchmarks and metrics. It details model architectures, parameter counts, and performance scores on a wide range of tasks including knowledge evaluation, language reasoning, code and math challenges, and long-context understanding. The document highlights the highest and second-best scores for each benchmark, emphasizing improvements across versions. Additionally, it describes the post-training phase for the DeepSeek-V4 series, noting a key methodological change from mixed Reinforcement Learning to On-Policy Distillation (OPD). The section on specialist training outlines the adaptation of the DeepSeek-V3.2 pipeline, involving fine-tuning followed by Reinforcement Learning using the Group Relative Policy Optimization (GRPO) algorithm with domain-specific prompts and rewards.", | |
| "text": "Table 1|Comparison among DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-\nPro-Base. All models are evaluated in our internal framework and share the same evaluation\nsetting. Scores with a gap not exceeding 0.3 are considered to be at the same level. The highest\nscore in each row is inbold font, and the second is underlined .\nBenchmark(Metric)# ShotsDeepSeek-V3.2 DeepSeek-V4-Flash DeepSeek-V4-Pro\nBase Base Base\nArchitecture - MoE MoE MoE\n# Activated Params - 37B 13B 49B\n# Total Params - 671B 284B 1.6T\nWorld Knowl.AGIEval(EM)0-shot 80.1 82.6 83.1\nMMLU(EM)5-shot 87.8 88.7 90.1\nMMLU-Redux(EM)5-shot 87.5 89.4 90.8\nMMLU-Pro(EM)5-shot 65.5 68.3 73.5\nMMMLU(EM)5-shot 87.9 88.8 90.3\nC-Eval(EM)5-shot 90.4 92.1 93.1\nCMMLU(EM)5-shot 88.9 90.4 90.8\nMultiLoKo(EM)5-shot 38.7 42.2 51.1\nSimple-QA verified(EM)25-shot 28.3 30.1 55.2\nSuperGPQA(EM)5-shot 45.0 46.5 53.9\nFACTS Parametric(EM)25-shot 27.1 33.9 62.6\nTriviaQA(EM)5-shot 83.3 82.885.6\nLang. & Reas.BBH(EM)3-shot 87.686.9 87.5\nDROP(F1)1-shot 88.2 88.6 88.7\nHellaSwag(EM)0-shot 86.4 85.788.0\nWinoGrande(EM)0-shot 78.9 79.5 81.5\nCLUEWSC(EM)5-shot 83.5 82.285.2\nCode & MathBigCodeBench(Pass@1)3-shot 63.956.8 59.2\nHumanEval(Pass@1)0-shot 62.8 69.5 76.8\nGSM8K(EM)8-shot 91.1 90.892.6\nMATH(EM)4-shot 60.5 57.464.5\nMGSM(EM)8-shot 81.385.784.4\nCMath(EM)3-shot 92.6 93.690.9\nLong Context LongBench-V2(EM)1-shot 40.2 44.7 51.5\n5. Post-Training\n5.1. Post-Training Pipeline\nFollowing pre-training, we conducted a post-training phase to yield the final models of DeepSeek-\nV4 series. Although the training pipeline largely mirrored that of DeepSeek-V3.2, a critical\nmethodological substitution was made: the mixed Reinforcement Learning (RL) stage was\nentirely replaced by On-Policy Distillation (OPD; Gu et al., 2024; Lu and Lab, 2025).\n5.1.1. Specialist Training\nThe development of domain specialists was conducted by adapting the DeepSeek-V3.2 training\npipeline. Specifically, each model was sequentially optimized through an initial fine-tuning\nphase and subsequent Reinforcement Learning (RL) guided by domain-specific prompts and re-\nward signals. For the RL stage, we implemented the Group Relative Policy Optimization (GRPO)\nalgorithm, maintaining hyper-parameters closely aligned with our prior research (DeepSeek-AI,\n2025; DeepSeek-AI, 2025).\n28", | |
| "nodes": [ | |
| { | |
| "title": "Post-Training Pipeline", | |
| "node_id": "0033", | |
| "start_index": 28, | |
| "end_index": 28, | |
| "summary": "The partial document presents a comparative analysis of three DeepSeek model versions\u2014DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-Pro-Base\u2014evaluated under a consistent internal framework across various benchmarks and metrics. It details model architectures, parameter counts, and performance scores on diverse tasks including knowledge evaluation, language reasoning, code and math challenges, and long-context understanding. The document highlights that scores within a 0.3 gap are considered equivalent and emphasizes top-performing results. Additionally, it describes the post-training phase for the DeepSeek-V4 series, noting a key methodological change from mixed Reinforcement Learning to On-Policy Distillation (OPD). The section on specialist training outlines the adaptation of the DeepSeek-V3.2 pipeline involving fine-tuning and Reinforcement Learning guided by domain-specific prompts, employing the Group Relative Policy Optimization (GRPO) algorithm with hyperparameters consistent with prior research.", | |
| "text": "Table 1|Comparison among DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-\nPro-Base. All models are evaluated in our internal framework and share the same evaluation\nsetting. Scores with a gap not exceeding 0.3 are considered to be at the same level. The highest\nscore in each row is inbold font, and the second is underlined .\nBenchmark(Metric)# ShotsDeepSeek-V3.2 DeepSeek-V4-Flash DeepSeek-V4-Pro\nBase Base Base\nArchitecture - MoE MoE MoE\n# Activated Params - 37B 13B 49B\n# Total Params - 671B 284B 1.6T\nWorld Knowl.AGIEval(EM)0-shot 80.1 82.6 83.1\nMMLU(EM)5-shot 87.8 88.7 90.1\nMMLU-Redux(EM)5-shot 87.5 89.4 90.8\nMMLU-Pro(EM)5-shot 65.5 68.3 73.5\nMMMLU(EM)5-shot 87.9 88.8 90.3\nC-Eval(EM)5-shot 90.4 92.1 93.1\nCMMLU(EM)5-shot 88.9 90.4 90.8\nMultiLoKo(EM)5-shot 38.7 42.2 51.1\nSimple-QA verified(EM)25-shot 28.3 30.1 55.2\nSuperGPQA(EM)5-shot 45.0 46.5 53.9\nFACTS Parametric(EM)25-shot 27.1 33.9 62.6\nTriviaQA(EM)5-shot 83.3 82.885.6\nLang. & Reas.BBH(EM)3-shot 87.686.9 87.5\nDROP(F1)1-shot 88.2 88.6 88.7\nHellaSwag(EM)0-shot 86.4 85.788.0\nWinoGrande(EM)0-shot 78.9 79.5 81.5\nCLUEWSC(EM)5-shot 83.5 82.285.2\nCode & MathBigCodeBench(Pass@1)3-shot 63.956.8 59.2\nHumanEval(Pass@1)0-shot 62.8 69.5 76.8\nGSM8K(EM)8-shot 91.1 90.892.6\nMATH(EM)4-shot 60.5 57.464.5\nMGSM(EM)8-shot 81.385.784.4\nCMath(EM)3-shot 92.6 93.690.9\nLong Context LongBench-V2(EM)1-shot 40.2 44.7 51.5\n5. Post-Training\n5.1. Post-Training Pipeline\nFollowing pre-training, we conducted a post-training phase to yield the final models of DeepSeek-\nV4 series. Although the training pipeline largely mirrored that of DeepSeek-V3.2, a critical\nmethodological substitution was made: the mixed Reinforcement Learning (RL) stage was\nentirely replaced by On-Policy Distillation (OPD; Gu et al., 2024; Lu and Lab, 2025).\n5.1.1. Specialist Training\nThe development of domain specialists was conducted by adapting the DeepSeek-V3.2 training\npipeline. Specifically, each model was sequentially optimized through an initial fine-tuning\nphase and subsequent Reinforcement Learning (RL) guided by domain-specific prompts and re-\nward signals. For the RL stage, we implemented the Group Relative Policy Optimization (GRPO)\nalgorithm, maintaining hyper-parameters closely aligned with our prior research (DeepSeek-AI,\n2025; DeepSeek-AI, 2025).\n28", | |
| "nodes": [ | |
| { | |
| "title": "Specialist Training", | |
| "node_id": "0034", | |
| "start_index": 28, | |
| "end_index": 32, | |
| "summary": "The partial document primarily covers a detailed comparison of three DeepSeek model variants\u2014DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-Pro-Base\u2014across various benchmarks and evaluation metrics, highlighting differences in architecture, parameter counts, and performance scores. It then extensively discusses the post-training phase of the DeepSeek-V4 series, emphasizing the replacement of the mixed Reinforcement Learning stage with On-Policy Distillation (OPD) for final model refinement. The document elaborates on specialist training involving domain-specific fine-tuning and reinforcement learning guided by the Group Relative Policy Optimization (GRPO) algorithm, with a focus on optimizing models for different reasoning effort modes (\"Non-think,\" \"Think,\" and \"Think Max\") that vary in speed and depth of logical analysis. It introduces specialized response formats and system prompt instructions to support these modes. Additionally, the text describes the use of a Generative Reward Model (GRM) in place of traditional scalar reward models for reinforcement learning, enabling joint optimization of generative and evaluative capabilities with minimal human annotation. The document also details a new XML-based tool-call schema for model-tool interactions, designed to reduce errors and improve robustness, along with an improved interleaved thinking management strategy that preserves reasoning traces across conversation turns in tool-calling scenarios. Finally, it introduces the Quick Instruction mechanism, which uses special tokens embedded in the input sequence to efficiently handle auxiliary chatbot tasks without redundant computation, thereby reducing response latency and engineering complexity.", | |
| "text": "Table 1|Comparison among DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base, and DeepSeek-V4-\nPro-Base. All models are evaluated in our internal framework and share the same evaluation\nsetting. Scores with a gap not exceeding 0.3 are considered to be at the same level. The highest\nscore in each row is inbold font, and the second is underlined .\nBenchmark(Metric)# ShotsDeepSeek-V3.2 DeepSeek-V4-Flash DeepSeek-V4-Pro\nBase Base Base\nArchitecture - MoE MoE MoE\n# Activated Params - 37B 13B 49B\n# Total Params - 671B 284B 1.6T\nWorld Knowl.AGIEval(EM)0-shot 80.1 82.6 83.1\nMMLU(EM)5-shot 87.8 88.7 90.1\nMMLU-Redux(EM)5-shot 87.5 89.4 90.8\nMMLU-Pro(EM)5-shot 65.5 68.3 73.5\nMMMLU(EM)5-shot 87.9 88.8 90.3\nC-Eval(EM)5-shot 90.4 92.1 93.1\nCMMLU(EM)5-shot 88.9 90.4 90.8\nMultiLoKo(EM)5-shot 38.7 42.2 51.1\nSimple-QA verified(EM)25-shot 28.3 30.1 55.2\nSuperGPQA(EM)5-shot 45.0 46.5 53.9\nFACTS Parametric(EM)25-shot 27.1 33.9 62.6\nTriviaQA(EM)5-shot 83.3 82.885.6\nLang. & Reas.BBH(EM)3-shot 87.686.9 87.5\nDROP(F1)1-shot 88.2 88.6 88.7\nHellaSwag(EM)0-shot 86.4 85.788.0\nWinoGrande(EM)0-shot 78.9 79.5 81.5\nCLUEWSC(EM)5-shot 83.5 82.285.2\nCode & MathBigCodeBench(Pass@1)3-shot 63.956.8 59.2\nHumanEval(Pass@1)0-shot 62.8 69.5 76.8\nGSM8K(EM)8-shot 91.1 90.892.6\nMATH(EM)4-shot 60.5 57.464.5\nMGSM(EM)8-shot 81.385.784.4\nCMath(EM)3-shot 92.6 93.690.9\nLong Context LongBench-V2(EM)1-shot 40.2 44.7 51.5\n5. Post-Training\n5.1. Post-Training Pipeline\nFollowing pre-training, we conducted a post-training phase to yield the final models of DeepSeek-\nV4 series. Although the training pipeline largely mirrored that of DeepSeek-V3.2, a critical\nmethodological substitution was made: the mixed Reinforcement Learning (RL) stage was\nentirely replaced by On-Policy Distillation (OPD; Gu et al., 2024; Lu and Lab, 2025).\n5.1.1. Specialist Training\nThe development of domain specialists was conducted by adapting the DeepSeek-V3.2 training\npipeline. Specifically, each model was sequentially optimized through an initial fine-tuning\nphase and subsequent Reinforcement Learning (RL) guided by domain-specific prompts and re-\nward signals. For the RL stage, we implemented the Group Relative Policy Optimization (GRPO)\nalgorithm, maintaining hyper-parameters closely aligned with our prior research (DeepSeek-AI,\n2025; DeepSeek-AI, 2025).\n28Reasoning Efforts.It is widely recognized that a model\u2019s performance on reasoning tasks\nis fundamentally governed by the computational effort expended. Consequently, we trained\ndistinct specialist models under divergent RL configurations to facilitate the development of\nmodels optimized for varying reasoning capacities. As detailed in Table 2, DeepSeek-V4-Pro and\nDeepSeek-V4-Flash both support three specific reasoning effort modes. For each mode, we apply\ndistinct length penalties and context windows during RL training, which results in varying\noutput token lengths for reasoning. To integrate these distinct reasoning modes, we utilize\nspecialized response formats demarcated by the <think> and</think> tokens. Furthermore,\nfor the \"Think Max\" mode, we prepend a specific instruction to the beginning of the system\nprompt to guide the model\u2019s reasoning process, as shown in Table 3.\nTable 2|Comparison of three reasoning modes\nReasoning\nModeCharacteristics Typical Use Cases Response Format\nNon-think Fast, intuitive re-\nsponses based on\nhabits or simple\nrules.Routine daily tasks,\nemergency reactions,\nlow-risk decisions.</think> summary\nThink High Conscious logical\nanalysis, slower but\nmore accurate.Complex problem-\nsolving, planning,\nmedium-risk deci-\nsions.<think> thinking\ntokens </think>\nsummary\nThink Max Push reasoning to its\nfullest extent. Slow\nbut powerful.Exploring the bound-\nary of model reason-\ning capability.1. A special system\nprompt at the begin-\nning.\n2.<think> thinking\ntokens </think>\nsummary\nTable 3|Instruction injected into the system prompt for the \"Think Max\" mode.\nInjected Instruction\nReasoning Effort: Absolute maximum with no shortcuts permitted.\nYou MUST be very thorough in your thinking and comprehensively decompose the\nproblem to resolve the root cause, rigorously stress-testing your logic against all potential\npaths, edge cases, and adversarial scenarios.\nExplicitly write out your entire deliberation process, documenting every intermediate\nstep, considered alternative, and rejected hypothesis to ensure absolutely no assumption\nis left unchecked.\nGenerative Reward Model.Typically, easy-to-verify tasks can be effectively optimized using\nsimple rule-based verifiers or test cases. In contrast, hard-to-verify tasks traditionally rely on\nReinforcement Learning from Human Feedback (RLHF), which necessitates extensive human\nannotation to train a scalar reward model. In the post-training phase of DeepSeek-V4 series,\n29however, we dispense with these conventional scalar-based reward models. Instead, to address\nhard-to-verify tasks, we curate rubric-guided RL data and employ a Generative Reward Model\n(GRM) to evaluate policy trajectories. Crucially, we apply RL optimization directly to the GRM\nitself. In this paradigm, the actor network natively functions as the GRM, enabling the joint\noptimization of the model\u2019s evaluative (judging) proficiency alongside its standard generative\ncapabilities. By unifying these roles, the model\u2019s internal reasoning capabilities are inherently\nfused into its evaluative process, resulting in highly robust scoring. Furthermore, this approach\nachieves superior performance with only a minimal set of diverse human annotations, as the\nmodel leverages its own logic to generalize across complex tasks.\nTable 4|Tool-call schema for DeepSeek-V4 series.\nTool Call Schema\n## Tools\nYou have access to a set of tools to help answer the user\u2019s question. You can\ninvoke tools by writing a \"<|DSML|tool_calls>\" block like the following:\n<|DSML|tool_calls>\n<|DSML|invoke name=\"$TOOL_NAME\">\n<|DSML|parameter name=\"$PARAMETER_NAME\" string=\"true|false\">$PARAMETER_VALUE\n</|DSML|parameter>\n...\n</|DSML|invoke>\n<|DSML|invoke name=\"$TOOL_NAME2\">\n...\n</|DSML|invoke>\n</|DSML|tool_calls>\nString parameters should be specified as is and set \u2018string=\"true\"\u2018. For all\nother types (numbers, booleans, arrays, objects), pass the value in JSON\nformat and set \u2018string=\"false\"\u2018.\nIf thinking_mode is enabled (triggered by <think>), you MUST output your\ncomplete reasoning inside <think>...</think> BEFORE any tool calls or\nfinal response.\nOtherwise, output directly after </think> with tool calls or final response.\n### Available Tool Schemas\n{Tool Definition...}\nYou MUST strictly follow the above definedtool name and parameter schemas to\ninvoke tool calls.\nTool-Call Schema and Special Token.Consistent with our previous version, we utilize a\ndedicated <think></think> tag to delineate the reasoning path. In DeepSeek-V4 series, we\nintroduce a new tool-call schema that employs a special \"|DSML|\" token and utilizes an XML-\nbased format for tool invocations, as demonstrated in Table 4. Our experiments demonstrate that\nthe XML format effectively mitigates escaping failures and reduces tool-call errors, providing a\nmore robust interface for model-tool interactions.\n30a)Thinkingwithtools\nb)ThinkingwithouttoolsFigure 7|Thinking management of DeepSeek-V4 series.\nInterleaved Thinking.DeepSeek-V3.2 introduced a context management strategy that retains\nreasoning traces across tool-result rounds but discards them upon the arrival of new user mes-\nsages. While effective, this still caused unnecessary token waste in complex agentic workflows\n\u2014 each new user turn would flush all accumulated reasoning content, forcing the model to\nreconstruct its problem-solving state from scratch. Leveraging the expanded 1M-token context\nwindow of DeepSeek-V4 series, we further refine this mechanism to maximize the effectiveness\nof interleaved thinking in agentic environments:\n\u2022Tool-Calling Scenarios.As illustrated in Figure 7(a), all reasoning content is fully pre-\nserved throughout the entire conversation. Unlike DeepSeek-V3.2, which discarded\nthinking traces upon each new user turn, DeepSeek-V4 series retain the complete reason-\ning history across all rounds, including across user message boundaries. This allows the\nmodel to maintain a coherent, cumulative chain of thought over long-horizon agent tasks.\n\u2022General Conversational Scenarios.As illustrated in Figure 7(b), the original strategy is\npreserved: reasoning content from previous turns is discarded when a new user message\narrives, keeping the context concise for settings where persistent reasoning traces provide\nlimited benefit.\nAs with DeepSeek-V3.2, agent frameworks that simulate tool interactions via user messages (e.g.,\nTerminus) may not trigger the tool-calling context path and thus may not benefit from enhanced\n31reasoning persistence. We continue to recommend non-think models for such architectures.\nTable 5|Quick Instruction special tokens for auxiliary tasks.\nSpecial Token Description Format\n<|action|>Determines whether the user\nprompt requires a web search\nor can be answered directly....<|User|>{prompt}<|Assistant|>\n<think><|action|>\n<|title|> Generates a concise conversa-\ntion title after the first assis-\ntant response....<|Assistant|>{response}\n<|end_of_sentence|><|title|>\n<|query|> Generates search queries for\nthe user prompt....<|User|>{prompt}<|query|>\n<|authority|> Classifies the user prompt\u2019s\ndemand for source authorita-\ntiveness....<|User|>{prompt}<|authority|>\n<|domain|> Identifies the domain of the\nuser prompt....<|User|>{prompt}<|domain|>\n<|extracted_url|>\n<|read_url|>Determines whether each\nURL in the user prompt\nshould be fetched and read....<|User|>{prompt}\n<|extracted_url|>{url}\n<|read_url|>\nQuick Instruction.In chatbot scenarios, a number of auxiliary tasks (e.g., determining whether\nto trigger a web search, intent recognition, etc.) must be executed before generating the response.\nConventionally, these tasks are handled by a separate small model, requiring redundant prefill-\ning since it cannot reuse the existing KV cache. To overcome this limitation, we introduce Quick\nInstruction. We append a set of dedicated special tokens directly to the input sequence, where\neach token corresponds to a specific auxiliary task. By directly reusing the already-computed\nKV cache, this mechanism completely avoids redundant prefilling and allows certain tasks, such\nas generating search queries and determining authority and domain, to be executed in parallel.\nConsequently, this approach significantly reduces the user-perceived time-to-first-token (TTFT)\nand eliminates the engineering overhead of maintaining and iterating an extra small model. The\nsupported Quick Instruction tokens are summarized in Table 5.\n5.1.2. On-Policy Distillation\nAfter training multiple domain-specific experts via specialized fine-tuning and reinforcement\nlearning, we employ multi-teacher On-Policy Distillation (OPD; Gu et al. 2024; Lu and Lab 2025)\nas the primary technique for merging expert capabilities into the final model. OPD has emerged\nas an effective post-training paradigm for efficiently transferring the knowledge and capabilities\nof domain experts to a single, unified model. This is achieved by having the student learn from\nthe output distributions of teacher models on its own generated trajectories. Formally, given a\nset of\ud835\udc41expert models{\ud835\udf0b \ud835\udc381,\ud835\udf0b\ud835\udc382,...,\ud835\udf0b\ud835\udc38\ud835\udc41}, the OPD objective function is defined as:\nLOPD(\ud835\udf03)=\ud835\udc41\u2211\ufe01\n\ud835\udc56=1\ud835\udc64\ud835\udc56\u00b7DKL\u0000\n\ud835\udf0b\ud835\udf03\u2225\ud835\udf0b\ud835\udc38\ud835\udc56\u0001. (29)\nIn this formulation,\ud835\udc64 \ud835\udc56represents the assigned weight for each expert, typically determined by\nthe relative importance of the expert. Computing the reverse KL loss DKL\u0000\n\ud835\udf0b\ud835\udf03\u2225\ud835\udf0b\ud835\udc38\ud835\udc56\u0001requires\n32" | |
| }, | |
| { | |
| "title": "On-Policy Distillation", | |
| "node_id": "0035", | |
| "start_index": 32, | |
| "end_index": 33, | |
| "summary": "The partial document discusses advanced techniques and infrastructure improvements for enhancing large language model training and deployment. It introduces \"Quick Instruction,\" a method using special tokens appended to input sequences to efficiently handle auxiliary chatbot tasks (e.g., web search triggering, intent recognition) by reusing the KV cache, thereby reducing latency and engineering complexity. The document then details \"On-Policy Distillation\" (OPD), a multi-teacher knowledge distillation approach that merges multiple domain-specific expert models into a single unified model by aligning output distributions on the student\u2019s own generated trajectories, using full-vocabulary logit distillation for improved training stability. Additionally, it covers post-training infrastructure enhancements, including scalable distributed training and rollout engines to support ultra-long-context reinforcement learning and OPD with many teacher models. A key optimization described is FP4 Quantization-Aware Training (QAT), applied to Mixture-of-Experts (MoE) weights and attention mechanisms to accelerate inference and reduce memory usage while maintaining accuracy, leveraging a lossless FP4-to-FP8 dequantization scheme and achieving significant speedups in key components.", | |
| "text": "reasoning persistence. We continue to recommend non-think models for such architectures.\nTable 5|Quick Instruction special tokens for auxiliary tasks.\nSpecial Token Description Format\n<|action|>Determines whether the user\nprompt requires a web search\nor can be answered directly....<|User|>{prompt}<|Assistant|>\n<think><|action|>\n<|title|> Generates a concise conversa-\ntion title after the first assis-\ntant response....<|Assistant|>{response}\n<|end_of_sentence|><|title|>\n<|query|> Generates search queries for\nthe user prompt....<|User|>{prompt}<|query|>\n<|authority|> Classifies the user prompt\u2019s\ndemand for source authorita-\ntiveness....<|User|>{prompt}<|authority|>\n<|domain|> Identifies the domain of the\nuser prompt....<|User|>{prompt}<|domain|>\n<|extracted_url|>\n<|read_url|>Determines whether each\nURL in the user prompt\nshould be fetched and read....<|User|>{prompt}\n<|extracted_url|>{url}\n<|read_url|>\nQuick Instruction.In chatbot scenarios, a number of auxiliary tasks (e.g., determining whether\nto trigger a web search, intent recognition, etc.) must be executed before generating the response.\nConventionally, these tasks are handled by a separate small model, requiring redundant prefill-\ning since it cannot reuse the existing KV cache. To overcome this limitation, we introduce Quick\nInstruction. We append a set of dedicated special tokens directly to the input sequence, where\neach token corresponds to a specific auxiliary task. By directly reusing the already-computed\nKV cache, this mechanism completely avoids redundant prefilling and allows certain tasks, such\nas generating search queries and determining authority and domain, to be executed in parallel.\nConsequently, this approach significantly reduces the user-perceived time-to-first-token (TTFT)\nand eliminates the engineering overhead of maintaining and iterating an extra small model. The\nsupported Quick Instruction tokens are summarized in Table 5.\n5.1.2. On-Policy Distillation\nAfter training multiple domain-specific experts via specialized fine-tuning and reinforcement\nlearning, we employ multi-teacher On-Policy Distillation (OPD; Gu et al. 2024; Lu and Lab 2025)\nas the primary technique for merging expert capabilities into the final model. OPD has emerged\nas an effective post-training paradigm for efficiently transferring the knowledge and capabilities\nof domain experts to a single, unified model. This is achieved by having the student learn from\nthe output distributions of teacher models on its own generated trajectories. Formally, given a\nset of\ud835\udc41expert models{\ud835\udf0b \ud835\udc381,\ud835\udf0b\ud835\udc382,...,\ud835\udf0b\ud835\udc38\ud835\udc41}, the OPD objective function is defined as:\nLOPD(\ud835\udf03)=\ud835\udc41\u2211\ufe01\n\ud835\udc56=1\ud835\udc64\ud835\udc56\u00b7DKL\u0000\n\ud835\udf0b\ud835\udf03\u2225\ud835\udf0b\ud835\udc38\ud835\udc56\u0001. (29)\nIn this formulation,\ud835\udc64 \ud835\udc56represents the assigned weight for each expert, typically determined by\nthe relative importance of the expert. Computing the reverse KL loss DKL\u0000\n\ud835\udf0b\ud835\udf03\u2225\ud835\udf0b\ud835\udc38\ud835\udc56\u0001requires\n32sampling training trajectories from the student \ud835\udf0b\ud835\udf03to maintain on-policy learning. The underly-\ning logic ensures that the unified policy \ud835\udf0b\ud835\udf03selectively learns from the specialized expert relevant\nto the current task context (e.g., aligning with the mathematics expert for math reasoning tasks\nand the coding expert for programming tasks). Through this mechanism, the knowledge from\nphysically distinct expert weights is consolidated into a unified parameter space via logits-level\nalignment, practically circumventing the performance degradation often encountered in tra-\nditional weight-merging or mixed RL techniques. In this stage, more than ten teacher models\ncovering various domains are employed to distill a single student model.\nIn handling the above OPD objective, prior works usually simplify the full-vocabulary KL\nloss into a token-level KL estimate at each token position, and reuse RL framework by replac-\ningsg\u0002\nlog\ud835\udf0b\ud835\udc38\ud835\udc56(\ud835\udc66\ud835\udc61|\ud835\udc65,\ud835\udc66<\ud835\udc61)\n\ud835\udf0b\ud835\udf03(\ud835\udc66\ud835\udc61|\ud835\udc65,\ud835\udc66<\ud835\udc61)\u0003\n(sgrepresents the stop gradient operation) as the per-token advantage\nestimate in the policy loss calculation. Although this approach is resource-efficient, it leads\nto high variance in gradient estimation and often causes training instability. Therefore, we\nadopt full-vocabulary logit distillation in our OPD. Preserving the complete logit distribution in\ncalculating reverse KL loss yields more stable gradient estimates and ensures faithful distillation\nof the teachers\u2019 knowledge. In the following subsection, we describe the engineering efforts that\nmake full-vocabulary OPD feasible at scale.\n5.2. Post-Training Infrastructures\nOur post-training infrastructure is built upon the scalable framework developed for DeepSeek-\nV3.2. Specifically, we integrate the same distributed training stack described in Section 3.4 and\nthe rollout engine introduced earlier for efficient auto-regressive sampling. Building on this\nfoundation, we introduce the following principal enhancements in the present work. These\ndesigns enable efficient execution of ultra-long-context RL and OPD merging tasks involving\nover ten distinct teacher models, thereby substantially accelerating the iteration cycle for model\nreleases.\n5.2.1. FP4 Quantization-Aware Training\nTo achieve inference acceleration and reducing memory traffic at deployment, we introduce\nQuantization-Aware Training (QAT) (Jacob et al., 2018) during the post-training stage, enabling\nthe model, including those of teacher and reference models, to adapt to the precision degradation\nintroduced by quantization. We apply FP4 (MXFP4) quantization (Rouhani et al., 2023) to two\ncomponents: (1) MoE expert weights, which are a major source of GPU memory occupancy (Ope-\nnAI, 2025), and (2) the Query-Key (QK) path in the indexer of CSA, where QK activations are\ncached, loaded, and multiplied entirely in FP4, accelerating attention score computation in\nlong-context scenarios. In addition, we further quantize the index scores \ud835\udc3c:,:from FP32 to BF16\nduring this QAT process. This optimization achieves a 2 \u00d7speedup for the top-k selector, while\npreserving a 99.7% recall rate of KV entries.\nFor MoE expert weights, following the common practice of QAT, the FP32 master weights\nmaintained by the optimizer are first quantized to FP4, then dequantized back to FP8 for\ncomputation. Notably, our FP4-to-FP8 dequantization is lossless. This is because FP8 (E4M3)\nhas 2 additional exponent bits compared with FP4 (E2M1), offering a larger dynamic range.\nConsequently, as long as the ratio between the maximum and minimum scale factors of the FP4\nsub-blocks (1\u00d732 tiles) within each FP8 quantization block (128 \u00d7128 tiles) does not exceed\na certain threshold, the fine-grained scale information can be fully absorbed by the extended\ndynamic range of FP8. We empirically verify that current weights satisfy this condition. This\n33" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Post-Training Infrastructures", | |
| "node_id": "0036", | |
| "start_index": 33, | |
| "end_index": 33, | |
| "summary": "The partial document discusses advanced techniques for training and optimizing a unified student model that learns from multiple specialized teacher models across various domains. It covers the use of on-policy learning by sampling training trajectories from the student policy to selectively distill knowledge from relevant experts, avoiding performance degradation common in traditional weight-merging methods. The document highlights the adoption of full-vocabulary logit distillation instead of token-level KL loss to achieve more stable gradient estimates and faithful knowledge transfer. Additionally, it details the post-training infrastructure enhancements, including scalable distributed training and rollout engines, enabling efficient handling of ultra-long-context reinforcement learning and merging tasks involving over ten teacher models. A key optimization introduced is FP4 quantization-aware training (QAT) applied to MoE expert weights and attention mechanisms, which accelerates inference and reduces memory usage while maintaining high accuracy. The document explains the technical approach to FP4-to-FP8 dequantization, ensuring lossless computation and significant speedups in model execution.", | |
| "text": "sampling training trajectories from the student \ud835\udf0b\ud835\udf03to maintain on-policy learning. The underly-\ning logic ensures that the unified policy \ud835\udf0b\ud835\udf03selectively learns from the specialized expert relevant\nto the current task context (e.g., aligning with the mathematics expert for math reasoning tasks\nand the coding expert for programming tasks). Through this mechanism, the knowledge from\nphysically distinct expert weights is consolidated into a unified parameter space via logits-level\nalignment, practically circumventing the performance degradation often encountered in tra-\nditional weight-merging or mixed RL techniques. In this stage, more than ten teacher models\ncovering various domains are employed to distill a single student model.\nIn handling the above OPD objective, prior works usually simplify the full-vocabulary KL\nloss into a token-level KL estimate at each token position, and reuse RL framework by replac-\ningsg\u0002\nlog\ud835\udf0b\ud835\udc38\ud835\udc56(\ud835\udc66\ud835\udc61|\ud835\udc65,\ud835\udc66<\ud835\udc61)\n\ud835\udf0b\ud835\udf03(\ud835\udc66\ud835\udc61|\ud835\udc65,\ud835\udc66<\ud835\udc61)\u0003\n(sgrepresents the stop gradient operation) as the per-token advantage\nestimate in the policy loss calculation. Although this approach is resource-efficient, it leads\nto high variance in gradient estimation and often causes training instability. Therefore, we\nadopt full-vocabulary logit distillation in our OPD. Preserving the complete logit distribution in\ncalculating reverse KL loss yields more stable gradient estimates and ensures faithful distillation\nof the teachers\u2019 knowledge. In the following subsection, we describe the engineering efforts that\nmake full-vocabulary OPD feasible at scale.\n5.2. Post-Training Infrastructures\nOur post-training infrastructure is built upon the scalable framework developed for DeepSeek-\nV3.2. Specifically, we integrate the same distributed training stack described in Section 3.4 and\nthe rollout engine introduced earlier for efficient auto-regressive sampling. Building on this\nfoundation, we introduce the following principal enhancements in the present work. These\ndesigns enable efficient execution of ultra-long-context RL and OPD merging tasks involving\nover ten distinct teacher models, thereby substantially accelerating the iteration cycle for model\nreleases.\n5.2.1. FP4 Quantization-Aware Training\nTo achieve inference acceleration and reducing memory traffic at deployment, we introduce\nQuantization-Aware Training (QAT) (Jacob et al., 2018) during the post-training stage, enabling\nthe model, including those of teacher and reference models, to adapt to the precision degradation\nintroduced by quantization. We apply FP4 (MXFP4) quantization (Rouhani et al., 2023) to two\ncomponents: (1) MoE expert weights, which are a major source of GPU memory occupancy (Ope-\nnAI, 2025), and (2) the Query-Key (QK) path in the indexer of CSA, where QK activations are\ncached, loaded, and multiplied entirely in FP4, accelerating attention score computation in\nlong-context scenarios. In addition, we further quantize the index scores \ud835\udc3c:,:from FP32 to BF16\nduring this QAT process. This optimization achieves a 2 \u00d7speedup for the top-k selector, while\npreserving a 99.7% recall rate of KV entries.\nFor MoE expert weights, following the common practice of QAT, the FP32 master weights\nmaintained by the optimizer are first quantized to FP4, then dequantized back to FP8 for\ncomputation. Notably, our FP4-to-FP8 dequantization is lossless. This is because FP8 (E4M3)\nhas 2 additional exponent bits compared with FP4 (E2M1), offering a larger dynamic range.\nConsequently, as long as the ratio between the maximum and minimum scale factors of the FP4\nsub-blocks (1\u00d732 tiles) within each FP8 quantization block (128 \u00d7128 tiles) does not exceed\na certain threshold, the fine-grained scale information can be fully absorbed by the extended\ndynamic range of FP8. We empirically verify that current weights satisfy this condition. This\n33", | |
| "nodes": [ | |
| { | |
| "title": "FP4 Quantization-Aware Training", | |
| "node_id": "0037", | |
| "start_index": 33, | |
| "end_index": 34, | |
| "summary": "The partial document discusses advanced techniques for training and optimizing large language models through on-policy distillation (OPD) and reinforcement learning (RL). It highlights a unified policy approach where a student model selectively learns from multiple specialized expert teacher models across different domains, consolidating their knowledge via logits-level alignment to avoid performance degradation common in traditional weight merging. The document critiques prior token-level KL loss approximations for causing training instability and instead adopts full-vocabulary logit distillation to achieve more stable and faithful knowledge transfer.\n\nIt further details the post-training infrastructure enhancements enabling scalable and efficient training with over ten teacher models, including:\n\n1. **FP4 Quantization-Aware Training (QAT):** Introducing FP4 quantization to reduce memory usage and accelerate inference, particularly for MoE expert weights and attention computations in long-context scenarios, while maintaining accuracy through a lossless FP4-to-FP8 dequantization scheme and leveraging existing FP8 training frameworks.\n\n2. **Efficient Teacher Scheduling:** Managing large-scale teacher models by offloading weights to distributed storage with ZeRO-like sharding, caching last-layer hidden states instead of full logits to reduce memory overhead, and asynchronously loading teacher prediction heads to minimize GPU memory footprint during training.\n\n3. **Preemptible and Fault-Tolerant Rollout Service:** Implementing a token-level Write-Ahead Log (WAL) system to enable interruption-resilient generation during RL/OPD rollouts, allowing seamless resumption after preemption or hardware failures without introducing length bias in generated sequences.\n\nOverall, the document covers methods to stabilize and scale the distillation of multiple expert models into a single student model, optimize memory and computation efficiency through quantization and smart scheduling, and ensure robust, fault-tolerant training workflows in large GPU clusters.", | |
| "text": "sampling training trajectories from the student \ud835\udf0b\ud835\udf03to maintain on-policy learning. The underly-\ning logic ensures that the unified policy \ud835\udf0b\ud835\udf03selectively learns from the specialized expert relevant\nto the current task context (e.g., aligning with the mathematics expert for math reasoning tasks\nand the coding expert for programming tasks). Through this mechanism, the knowledge from\nphysically distinct expert weights is consolidated into a unified parameter space via logits-level\nalignment, practically circumventing the performance degradation often encountered in tra-\nditional weight-merging or mixed RL techniques. In this stage, more than ten teacher models\ncovering various domains are employed to distill a single student model.\nIn handling the above OPD objective, prior works usually simplify the full-vocabulary KL\nloss into a token-level KL estimate at each token position, and reuse RL framework by replac-\ningsg\u0002\nlog\ud835\udf0b\ud835\udc38\ud835\udc56(\ud835\udc66\ud835\udc61|\ud835\udc65,\ud835\udc66<\ud835\udc61)\n\ud835\udf0b\ud835\udf03(\ud835\udc66\ud835\udc61|\ud835\udc65,\ud835\udc66<\ud835\udc61)\u0003\n(sgrepresents the stop gradient operation) as the per-token advantage\nestimate in the policy loss calculation. Although this approach is resource-efficient, it leads\nto high variance in gradient estimation and often causes training instability. Therefore, we\nadopt full-vocabulary logit distillation in our OPD. Preserving the complete logit distribution in\ncalculating reverse KL loss yields more stable gradient estimates and ensures faithful distillation\nof the teachers\u2019 knowledge. In the following subsection, we describe the engineering efforts that\nmake full-vocabulary OPD feasible at scale.\n5.2. Post-Training Infrastructures\nOur post-training infrastructure is built upon the scalable framework developed for DeepSeek-\nV3.2. Specifically, we integrate the same distributed training stack described in Section 3.4 and\nthe rollout engine introduced earlier for efficient auto-regressive sampling. Building on this\nfoundation, we introduce the following principal enhancements in the present work. These\ndesigns enable efficient execution of ultra-long-context RL and OPD merging tasks involving\nover ten distinct teacher models, thereby substantially accelerating the iteration cycle for model\nreleases.\n5.2.1. FP4 Quantization-Aware Training\nTo achieve inference acceleration and reducing memory traffic at deployment, we introduce\nQuantization-Aware Training (QAT) (Jacob et al., 2018) during the post-training stage, enabling\nthe model, including those of teacher and reference models, to adapt to the precision degradation\nintroduced by quantization. We apply FP4 (MXFP4) quantization (Rouhani et al., 2023) to two\ncomponents: (1) MoE expert weights, which are a major source of GPU memory occupancy (Ope-\nnAI, 2025), and (2) the Query-Key (QK) path in the indexer of CSA, where QK activations are\ncached, loaded, and multiplied entirely in FP4, accelerating attention score computation in\nlong-context scenarios. In addition, we further quantize the index scores \ud835\udc3c:,:from FP32 to BF16\nduring this QAT process. This optimization achieves a 2 \u00d7speedup for the top-k selector, while\npreserving a 99.7% recall rate of KV entries.\nFor MoE expert weights, following the common practice of QAT, the FP32 master weights\nmaintained by the optimizer are first quantized to FP4, then dequantized back to FP8 for\ncomputation. Notably, our FP4-to-FP8 dequantization is lossless. This is because FP8 (E4M3)\nhas 2 additional exponent bits compared with FP4 (E2M1), offering a larger dynamic range.\nConsequently, as long as the ratio between the maximum and minimum scale factors of the FP4\nsub-blocks (1\u00d732 tiles) within each FP8 quantization block (128 \u00d7128 tiles) does not exceed\na certain threshold, the fine-grained scale information can be fully absorbed by the extended\ndynamic range of FP8. We empirically verify that current weights satisfy this condition. This\n33allows the entire QAT pipeline to fully reuse the existing FP8 training framework without\nany modification. In the backward pass, gradients are computed with respect to the same FP8\nweights in the forward pass and directly propagated back to the FP32 master weights, equivalent\nto applying the Straight-Through Estimator (STE) through the quantization operation. This also\navoids the need to re-quantize transposed weights.\nDuring the inference and rollout phases of RL training, which do not involve backward\npasses, we directly use native FP4 quantized weights instead of simulated quantization. This\nensures that model behavior during sampling is fully consistent with online deployment, while\nalso reducing kernel memory loading for actual speedup and significantly lowering memory\nconsumption. We process the QK path in the indexer of CSA similarly.\n5.2.2. Efficient Teacher Scheduling for Full-Vocabulary OPD\nOur framework supports full-vocabulary On-Policy Distillation (OPD) with an effectively\nunbounded number of teachers, each potentially comprising trillions of parameters. To enable\nthis, all teacher weights are offloaded to a centralized distributed storage and are loaded on\ndemand during the teacher forward pass with ZeRO-like parameter sharding to alleviate both\nI/O and DRAM pressure. Furthermore, naively materializing logits for a vocabulary size\n|\ud835\udc49|> 100kacross all teachers is prohibitive, even when spooled to disk. We address this by\ncaching only the last-layer teacher hidden states in a centralized buffer during the forward\npass. At training time, these cached states are retrieved and passed through the corresponding\nprediction head module to reconstruct the full logits on the fly. This design incurs negligible\nrecomputation overhead while completely circumventing the memory burden associated with\nexplicit logits materialization. To mitigate the GPU memory footprint of the teacher prediction\nhead, we order training samples by teacher index during data dispatching. This arrangement\nensures that each distinct teacher head is loaded only once per mini-batch and that at most\none teacher head resides in device memory at any given time. All parameters and hidden state\nloading/offloading operations proceed asynchronously in the background, without blocking\ncomputation on the critical path. Finally, the exact KL divergences between teacher and student\nlogits are computed using a specialized TileLang kernel, which accelerates the computation and\ncurtails dynamic memory allocation.\n5.2.3. Preemptible and Fault-Tolerant Rollout Service\nTo maximize GPU resource utilization while enabling rapid hardware provisioning for high-\npriority tasks, our GPU cluster employs a cluster-wide preemptive task scheduler, where any\nrunning task may be preempted at any time. Also, hardware failures are prevalent in large-scale\nGPU clusters. To this end, we implement a preemptible and fault-tolerant LLM generation\nservice for RL/OPD rollout.\nSpecifically, we implement a token-granular Write-Ahead Log (WAL) for each generation\nrequest. Whenever a new token is generated for a request, we immediately append it to that\nrequest\u2019s WAL. During preemption, we pause the inference engine and save the KV cache of\nunfinished requests. Upon resumption, we use the persisted WALs and saved KV cache to\ncontinue decoding. Even when a fatal hardware error occurs, we can re-run the prefill phase\nusing the persisted tokens in WAL to reconstruct the KV cache.\nImportantly, it is mathematically incorrect to regenerate unfinished requests from scratch,\nas this introduces length bias. Because shorter responses are more likely to survive interrup-\ntion, regenerating from scratch makes the model more prone to producing shorter sequences\n34" | |
| }, | |
| { | |
| "title": "Efficient Teacher Scheduling for Full-Vocabulary OPD", | |
| "node_id": "0038", | |
| "start_index": 34, | |
| "end_index": 34, | |
| "summary": "The partial document discusses advanced techniques for efficient quantized training and inference in large language models, focusing on FP8 and FP4 precision formats to optimize memory and computation. It details a method for backward gradient propagation using the Straight-Through Estimator to avoid re-quantization overhead. During inference and reinforcement learning rollout phases, native FP4 quantized weights are used to ensure consistency and reduce memory usage. The document also presents an efficient teacher scheduling framework for full-vocabulary On-Policy Distillation (OPD) that supports an unbounded number of large-scale teacher models by offloading weights to distributed storage and caching only last-layer hidden states to avoid prohibitive memory costs. It describes asynchronous loading strategies and optimized computation of KL divergences using specialized kernels. Additionally, it introduces a preemptible and fault-tolerant rollout service for large-scale GPU clusters, employing a token-granular Write-Ahead Log to enable seamless resumption of interrupted inference tasks, preventing length bias and ensuring robustness against hardware failures.", | |
| "text": "allows the entire QAT pipeline to fully reuse the existing FP8 training framework without\nany modification. In the backward pass, gradients are computed with respect to the same FP8\nweights in the forward pass and directly propagated back to the FP32 master weights, equivalent\nto applying the Straight-Through Estimator (STE) through the quantization operation. This also\navoids the need to re-quantize transposed weights.\nDuring the inference and rollout phases of RL training, which do not involve backward\npasses, we directly use native FP4 quantized weights instead of simulated quantization. This\nensures that model behavior during sampling is fully consistent with online deployment, while\nalso reducing kernel memory loading for actual speedup and significantly lowering memory\nconsumption. We process the QK path in the indexer of CSA similarly.\n5.2.2. Efficient Teacher Scheduling for Full-Vocabulary OPD\nOur framework supports full-vocabulary On-Policy Distillation (OPD) with an effectively\nunbounded number of teachers, each potentially comprising trillions of parameters. To enable\nthis, all teacher weights are offloaded to a centralized distributed storage and are loaded on\ndemand during the teacher forward pass with ZeRO-like parameter sharding to alleviate both\nI/O and DRAM pressure. Furthermore, naively materializing logits for a vocabulary size\n|\ud835\udc49|> 100kacross all teachers is prohibitive, even when spooled to disk. We address this by\ncaching only the last-layer teacher hidden states in a centralized buffer during the forward\npass. At training time, these cached states are retrieved and passed through the corresponding\nprediction head module to reconstruct the full logits on the fly. This design incurs negligible\nrecomputation overhead while completely circumventing the memory burden associated with\nexplicit logits materialization. To mitigate the GPU memory footprint of the teacher prediction\nhead, we order training samples by teacher index during data dispatching. This arrangement\nensures that each distinct teacher head is loaded only once per mini-batch and that at most\none teacher head resides in device memory at any given time. All parameters and hidden state\nloading/offloading operations proceed asynchronously in the background, without blocking\ncomputation on the critical path. Finally, the exact KL divergences between teacher and student\nlogits are computed using a specialized TileLang kernel, which accelerates the computation and\ncurtails dynamic memory allocation.\n5.2.3. Preemptible and Fault-Tolerant Rollout Service\nTo maximize GPU resource utilization while enabling rapid hardware provisioning for high-\npriority tasks, our GPU cluster employs a cluster-wide preemptive task scheduler, where any\nrunning task may be preempted at any time. Also, hardware failures are prevalent in large-scale\nGPU clusters. To this end, we implement a preemptible and fault-tolerant LLM generation\nservice for RL/OPD rollout.\nSpecifically, we implement a token-granular Write-Ahead Log (WAL) for each generation\nrequest. Whenever a new token is generated for a request, we immediately append it to that\nrequest\u2019s WAL. During preemption, we pause the inference engine and save the KV cache of\nunfinished requests. Upon resumption, we use the persisted WALs and saved KV cache to\ncontinue decoding. Even when a fatal hardware error occurs, we can re-run the prefill phase\nusing the persisted tokens in WAL to reconstruct the KV cache.\nImportantly, it is mathematically incorrect to regenerate unfinished requests from scratch,\nas this introduces length bias. Because shorter responses are more likely to survive interrup-\ntion, regenerating from scratch makes the model more prone to producing shorter sequences\n34" | |
| }, | |
| { | |
| "title": "Preemptible and Fault-Tolerant Rollout Service", | |
| "node_id": "0039", | |
| "start_index": 34, | |
| "end_index": 35, | |
| "summary": "The partial document discusses advanced techniques and system designs for efficient large-scale model training and deployment, particularly focusing on quantization-aware training (QAT), reinforcement learning (RL), and on-policy distillation (OPD). Key points include:\n\n1. **FP4 Quantization and Gradient Propagation:** The QAT pipeline reuses an existing FP8 training framework without modification, using Straight-Through Estimator (STE) for gradient computation and avoiding re-quantization of weights. During inference and rollout, native FP4 quantized weights are used to ensure consistency and reduce memory and computational overhead.\n\n2. **Efficient Teacher Scheduling for Full-Vocabulary OPD:** The framework supports distillation from an effectively unlimited number of large teacher models by offloading teacher weights to distributed storage and loading them on demand with ZeRO-like sharding. It avoids prohibitive memory costs by caching only last-layer hidden states and reconstructing logits on the fly, while asynchronously managing memory to keep only one teacher prediction head in GPU memory at a time. KL divergence computations are accelerated with a specialized kernel.\n\n3. **Preemptible and Fault-Tolerant Rollout Service:** To maximize GPU utilization and handle hardware failures, a token-granular Write-Ahead Log (WAL) system is implemented for generation requests, enabling interruption-resilient rollouts without introducing length bias. This approach is more efficient and mathematically correct compared to regenerating sequences from scratch.\n\n4. **Scaling RL Framework for Million-Token Contexts:** The rollout data is split into lightweight metadata and heavy per-token fields to optimize memory usage and data loading. Shared-memory loaders and dynamic mini-batch sizing reduce CPU/GPU memory pressure and balance throughput with I/O overlap.\n\n5. **Sandbox Infrastructure for Agentic AI (DeepSeek Elastic Compute - DSec):** A production-grade, horizontally scalable sandbox platform is introduced to support diverse, heterogeneous agentic AI workloads. DSec features a unified Python SDK abstracting four execution substrates (Function Call, Container, microVM, fullVM) with a common API, enabling flexible and secure execution environments. Fast image loading is achieved through layered, on-demand storage using 3FS-backed readonly EROFS layers, optimizing startup times and resource utilization.\n\nOverall, the document covers innovations in quantization, distributed training, fault tolerance, large-context RL, and flexible sandboxing infrastructure to support scalable, efficient, and robust AI model training and deployment.", | |
| "text": "allows the entire QAT pipeline to fully reuse the existing FP8 training framework without\nany modification. In the backward pass, gradients are computed with respect to the same FP8\nweights in the forward pass and directly propagated back to the FP32 master weights, equivalent\nto applying the Straight-Through Estimator (STE) through the quantization operation. This also\navoids the need to re-quantize transposed weights.\nDuring the inference and rollout phases of RL training, which do not involve backward\npasses, we directly use native FP4 quantized weights instead of simulated quantization. This\nensures that model behavior during sampling is fully consistent with online deployment, while\nalso reducing kernel memory loading for actual speedup and significantly lowering memory\nconsumption. We process the QK path in the indexer of CSA similarly.\n5.2.2. Efficient Teacher Scheduling for Full-Vocabulary OPD\nOur framework supports full-vocabulary On-Policy Distillation (OPD) with an effectively\nunbounded number of teachers, each potentially comprising trillions of parameters. To enable\nthis, all teacher weights are offloaded to a centralized distributed storage and are loaded on\ndemand during the teacher forward pass with ZeRO-like parameter sharding to alleviate both\nI/O and DRAM pressure. Furthermore, naively materializing logits for a vocabulary size\n|\ud835\udc49|> 100kacross all teachers is prohibitive, even when spooled to disk. We address this by\ncaching only the last-layer teacher hidden states in a centralized buffer during the forward\npass. At training time, these cached states are retrieved and passed through the corresponding\nprediction head module to reconstruct the full logits on the fly. This design incurs negligible\nrecomputation overhead while completely circumventing the memory burden associated with\nexplicit logits materialization. To mitigate the GPU memory footprint of the teacher prediction\nhead, we order training samples by teacher index during data dispatching. This arrangement\nensures that each distinct teacher head is loaded only once per mini-batch and that at most\none teacher head resides in device memory at any given time. All parameters and hidden state\nloading/offloading operations proceed asynchronously in the background, without blocking\ncomputation on the critical path. Finally, the exact KL divergences between teacher and student\nlogits are computed using a specialized TileLang kernel, which accelerates the computation and\ncurtails dynamic memory allocation.\n5.2.3. Preemptible and Fault-Tolerant Rollout Service\nTo maximize GPU resource utilization while enabling rapid hardware provisioning for high-\npriority tasks, our GPU cluster employs a cluster-wide preemptive task scheduler, where any\nrunning task may be preempted at any time. Also, hardware failures are prevalent in large-scale\nGPU clusters. To this end, we implement a preemptible and fault-tolerant LLM generation\nservice for RL/OPD rollout.\nSpecifically, we implement a token-granular Write-Ahead Log (WAL) for each generation\nrequest. Whenever a new token is generated for a request, we immediately append it to that\nrequest\u2019s WAL. During preemption, we pause the inference engine and save the KV cache of\nunfinished requests. Upon resumption, we use the persisted WALs and saved KV cache to\ncontinue decoding. Even when a fatal hardware error occurs, we can re-run the prefill phase\nusing the persisted tokens in WAL to reconstruct the KV cache.\nImportantly, it is mathematically incorrect to regenerate unfinished requests from scratch,\nas this introduces length bias. Because shorter responses are more likely to survive interrup-\ntion, regenerating from scratch makes the model more prone to producing shorter sequences\n34whenever an interruption occurs. If the inference stack is batch-invariant and deterministic,\nthis correctness issue could also be addressed by regenerating with a consistent seed for the\npseudorandom number generator used in the sampler. However, this approach still incurs the\nextra cost of re-running the decoding phase, making it far less efficient than our token-granular\nWAL method.\n5.2.4. Scaling RL Framework for Million-Token Context\nWe introduce targeted optimizations for efficient RL and OPD on million-token sequences.\nDuring the rollout phase, we adopt a preemptible and fault-tolerant rollout service, detailed in\nSection 5.2.3. For the inference and training phase, we decompose the rollout data format into\nlightweight metadata and heavy per-token fields. During data dispatching, the metadata for the\nentire rollout data can be loaded to perform global shuffling and packing layout computation.\nHeavy per-token fields are loaded via a shared-memory data loader to eliminate intra-node\ndata redundancy and are released immediately upon consumption at the mini-batch granularity,\nsubstantially reducing both CPU and GPU memory pressure. The number of on-device mini-\nbatches is dynamically determined based on workload, allowing an efficient trade-off between\ncomputational throughput and I/O overlap.\n5.2.5. Sandbox Infrastructure for Agentic AI\nTo meet the diverse execution demands of agentic AI during post-training and evaluation,\nwe build a production-grade sandbox platform,DeepSeek Elastic Compute (DSec). DSec\ncomprises three Rust components \u2014 the API gateway ( Apiserver ), per-host agent ( Edge ), and\nthe cluster monitor ( Watcher ) \u2014 that are interconnected by a custom RPC protocol and scale\nhorizontally atop the 3FS distributed filesystem (DeepSeek-AI, 2025). In production, a single\nDSec cluster manages hundreds of thousands of concurrent sandbox instances.\nThe design of DSec is motivated by four observations: (1) agentic workloads are highly\nheterogeneous, spanning lightweight function calls to full software-engineering pipelines with\ndiverse OS and security requirements; (2) environment images are numerous and large, yet must\nload quickly and support iterative customization; (3) high-density deployment demands efficient\nCPU and memory utilization; (4) sandbox lifecycles must coordinate with GPU training sched-\nules, including preemption and checkpoint-based resumption. Based on these observations, we\nelaborate on the four core designs of DSec individually in the following.\nFour Execution Substrates Behind One Unified Interface.DSec exposes a single Python\nSDK ( libdsec ) that abstracts four execution substrates.Function Calldispatches stateless\ninvocations to a pre-warmed container pool, eliminating cold-start overhead.Containeris fully\nDocker-compatible and leverages EROFS (Gao et al., 2019) on-demand loading for efficient\nimage assembly.microVM, built on Firecracker (Agache et al., 2020), adds VM-level isolation for\nsecurity-sensitive, high-density deployments.fullVM, built on QEMU (Bellard, 2005), supports\narbitrary guest operating systems. All four share a common API surface \u2014 command execution,\nfile transfer, and TTY access \u2014 and switching between them requires only a parameter change.\nFast Image Loading via Layered Storage.DSec reconciles fast startup with a large and growing\ncorpus of environment images through layered, on-demand loading. For containers, base images\nand filesystem commits are stored as 3FS-backed readonly EROFS layers mounted directly into\noverlay lowerdir s. We keep file metadata readily available on the local disk at mount time;\n35" | |
| }, | |
| { | |
| "title": "Scaling RL Framework for Million-Token Context", | |
| "node_id": "0040", | |
| "start_index": 35, | |
| "end_index": 35, | |
| "summary": "The partial document discusses advanced techniques and infrastructure for scaling reinforcement learning (RL) and agentic AI systems. It covers a token-granular Write-Ahead Logging (WAL) method to improve efficiency during interruptions in inference. It introduces optimizations for handling million-token sequences in RL, including a fault-tolerant rollout service, efficient data loading strategies that reduce CPU and GPU memory usage, and dynamic mini-batch sizing to balance throughput and I/O. The document also details the design and implementation of DeepSeek Elastic Compute (DSec), a production-grade sandbox platform for diverse agentic AI workloads. DSec features a unified Python SDK that supports four execution substrates\u2014Function Call, Container, microVM, and fullVM\u2014each providing different levels of isolation and compatibility. Additionally, it employs layered, on-demand image loading using a distributed filesystem and EROFS to enable fast startup and efficient management of large environment image corpora. The design addresses challenges such as heterogeneous workloads, rapid image loading, resource efficiency, and coordination with GPU training schedules.", | |
| "text": "whenever an interruption occurs. If the inference stack is batch-invariant and deterministic,\nthis correctness issue could also be addressed by regenerating with a consistent seed for the\npseudorandom number generator used in the sampler. However, this approach still incurs the\nextra cost of re-running the decoding phase, making it far less efficient than our token-granular\nWAL method.\n5.2.4. Scaling RL Framework for Million-Token Context\nWe introduce targeted optimizations for efficient RL and OPD on million-token sequences.\nDuring the rollout phase, we adopt a preemptible and fault-tolerant rollout service, detailed in\nSection 5.2.3. For the inference and training phase, we decompose the rollout data format into\nlightweight metadata and heavy per-token fields. During data dispatching, the metadata for the\nentire rollout data can be loaded to perform global shuffling and packing layout computation.\nHeavy per-token fields are loaded via a shared-memory data loader to eliminate intra-node\ndata redundancy and are released immediately upon consumption at the mini-batch granularity,\nsubstantially reducing both CPU and GPU memory pressure. The number of on-device mini-\nbatches is dynamically determined based on workload, allowing an efficient trade-off between\ncomputational throughput and I/O overlap.\n5.2.5. Sandbox Infrastructure for Agentic AI\nTo meet the diverse execution demands of agentic AI during post-training and evaluation,\nwe build a production-grade sandbox platform,DeepSeek Elastic Compute (DSec). DSec\ncomprises three Rust components \u2014 the API gateway ( Apiserver ), per-host agent ( Edge ), and\nthe cluster monitor ( Watcher ) \u2014 that are interconnected by a custom RPC protocol and scale\nhorizontally atop the 3FS distributed filesystem (DeepSeek-AI, 2025). In production, a single\nDSec cluster manages hundreds of thousands of concurrent sandbox instances.\nThe design of DSec is motivated by four observations: (1) agentic workloads are highly\nheterogeneous, spanning lightweight function calls to full software-engineering pipelines with\ndiverse OS and security requirements; (2) environment images are numerous and large, yet must\nload quickly and support iterative customization; (3) high-density deployment demands efficient\nCPU and memory utilization; (4) sandbox lifecycles must coordinate with GPU training sched-\nules, including preemption and checkpoint-based resumption. Based on these observations, we\nelaborate on the four core designs of DSec individually in the following.\nFour Execution Substrates Behind One Unified Interface.DSec exposes a single Python\nSDK ( libdsec ) that abstracts four execution substrates.Function Calldispatches stateless\ninvocations to a pre-warmed container pool, eliminating cold-start overhead.Containeris fully\nDocker-compatible and leverages EROFS (Gao et al., 2019) on-demand loading for efficient\nimage assembly.microVM, built on Firecracker (Agache et al., 2020), adds VM-level isolation for\nsecurity-sensitive, high-density deployments.fullVM, built on QEMU (Bellard, 2005), supports\narbitrary guest operating systems. All four share a common API surface \u2014 command execution,\nfile transfer, and TTY access \u2014 and switching between them requires only a parameter change.\nFast Image Loading via Layered Storage.DSec reconciles fast startup with a large and growing\ncorpus of environment images through layered, on-demand loading. For containers, base images\nand filesystem commits are stored as 3FS-backed readonly EROFS layers mounted directly into\noverlay lowerdir s. We keep file metadata readily available on the local disk at mount time;\n35" | |
| }, | |
| { | |
| "title": "Sandbox Infrastructure for Agentic AI", | |
| "node_id": "0041", | |
| "start_index": 35, | |
| "end_index": 36, | |
| "summary": "The partial document covers several key points related to optimizing reinforcement learning (RL) frameworks and infrastructure for large-scale AI workloads. It discusses a token-granular Write-Ahead Logging (WAL) method to improve efficiency during interruptions in inference. It introduces targeted optimizations for scaling RL and online policy distillation (OPD) to handle million-token sequences, including a fault-tolerant rollout service and memory-efficient data loading strategies that reduce CPU and GPU pressure. The document also details the design and implementation of DeepSeek Elastic Compute (DSec), a production-grade sandbox platform for agentic AI that supports diverse, heterogeneous workloads with four execution substrates unified under a single API. DSec features fast image loading via layered storage, density optimizations for massive concurrency, and trajectory logging for preemption-safe task resumption and deterministic replay. Finally, the document outlines the evaluation setup for benchmarking knowledge, reasoning, and code tasks using various datasets and describes the methodology for assessing model performance on Codeforces contests and other reasoning benchmarks with different context window sizes and temperature settings.", | |
| "text": "whenever an interruption occurs. If the inference stack is batch-invariant and deterministic,\nthis correctness issue could also be addressed by regenerating with a consistent seed for the\npseudorandom number generator used in the sampler. However, this approach still incurs the\nextra cost of re-running the decoding phase, making it far less efficient than our token-granular\nWAL method.\n5.2.4. Scaling RL Framework for Million-Token Context\nWe introduce targeted optimizations for efficient RL and OPD on million-token sequences.\nDuring the rollout phase, we adopt a preemptible and fault-tolerant rollout service, detailed in\nSection 5.2.3. For the inference and training phase, we decompose the rollout data format into\nlightweight metadata and heavy per-token fields. During data dispatching, the metadata for the\nentire rollout data can be loaded to perform global shuffling and packing layout computation.\nHeavy per-token fields are loaded via a shared-memory data loader to eliminate intra-node\ndata redundancy and are released immediately upon consumption at the mini-batch granularity,\nsubstantially reducing both CPU and GPU memory pressure. The number of on-device mini-\nbatches is dynamically determined based on workload, allowing an efficient trade-off between\ncomputational throughput and I/O overlap.\n5.2.5. Sandbox Infrastructure for Agentic AI\nTo meet the diverse execution demands of agentic AI during post-training and evaluation,\nwe build a production-grade sandbox platform,DeepSeek Elastic Compute (DSec). DSec\ncomprises three Rust components \u2014 the API gateway ( Apiserver ), per-host agent ( Edge ), and\nthe cluster monitor ( Watcher ) \u2014 that are interconnected by a custom RPC protocol and scale\nhorizontally atop the 3FS distributed filesystem (DeepSeek-AI, 2025). In production, a single\nDSec cluster manages hundreds of thousands of concurrent sandbox instances.\nThe design of DSec is motivated by four observations: (1) agentic workloads are highly\nheterogeneous, spanning lightweight function calls to full software-engineering pipelines with\ndiverse OS and security requirements; (2) environment images are numerous and large, yet must\nload quickly and support iterative customization; (3) high-density deployment demands efficient\nCPU and memory utilization; (4) sandbox lifecycles must coordinate with GPU training sched-\nules, including preemption and checkpoint-based resumption. Based on these observations, we\nelaborate on the four core designs of DSec individually in the following.\nFour Execution Substrates Behind One Unified Interface.DSec exposes a single Python\nSDK ( libdsec ) that abstracts four execution substrates.Function Calldispatches stateless\ninvocations to a pre-warmed container pool, eliminating cold-start overhead.Containeris fully\nDocker-compatible and leverages EROFS (Gao et al., 2019) on-demand loading for efficient\nimage assembly.microVM, built on Firecracker (Agache et al., 2020), adds VM-level isolation for\nsecurity-sensitive, high-density deployments.fullVM, built on QEMU (Bellard, 2005), supports\narbitrary guest operating systems. All four share a common API surface \u2014 command execution,\nfile transfer, and TTY access \u2014 and switching between them requires only a parameter change.\nFast Image Loading via Layered Storage.DSec reconciles fast startup with a large and growing\ncorpus of environment images through layered, on-demand loading. For containers, base images\nand filesystem commits are stored as 3FS-backed readonly EROFS layers mounted directly into\noverlay lowerdir s. We keep file metadata readily available on the local disk at mount time;\n35meanwhile, data blocks are fetched from 3FS upon request. For microVMs, DSec uses the\noverlaybd (Li et al., 2020) disk format: the read-only base layer resides on 3FS for cross-\ninstance sharing, while writes go to a local copy-on-write layer. Such snapshots are chainable,\nfacilitating efficient versioning and millisecond-scale resumption.\nDensity Optimizations Under Massive Concurrency.To accommodate hundreds of thousands\nof sandboxes per cluster, DSec tackles two resource bottlenecks. First, it mitigates duplicate\npage-cache footprints in virtualized environments and applies memory reclamation to enable\nsafe overcommitment. Second, it alleviates spinlock contention in the container runtime and\ntherefore, reduces per-sandbox CPU overhead, significantly increasing per-host packing density.\nTrajectory Logging and Preemption-Safe Resumption.DSec maintains a globally ordered\ntrajectory log for each sandbox, persistently recording every command invocation and its results.\nThe trajectory serves three purposes: (1)client fast-forwarding\u2014 when a training task is\npreempted, sandbox resources are retained nonetheless; upon resumption, DSec replays cached\nresults for previously completed commands, accelerating task recovery whilst also preventing\nerrors from re-execution of non-idempotent operations; (2)fine-grained provenance\u2014 the\norigin and corresponding outcomes of each state change are traceable; (3)deterministic replay\n\u2014 any historical session can be faithfully reproduced from its trajectory.\n5.3. Standard Benchmark Evaluation\n5.3.1. Evaluation Setup\nKnowledge and Reasoning.Knowledge and reasoning datasets include MMLU-Pro (Wang\net al., 2024b), GPQA (Rein et al., 2023), Human Last Exam (Phan et al., 2025), Simple-QA Veri-\nfied (Haas et al., 2025), Chinese-SimpleQA (He et al., 2024), LiveCodeBench-v6 (Jain et al., 2024),\nCodeForces (Internal Benchmark), HMMT 2026 Feb, Apex (Balunovi\u00b4 c et al., 2025), Apex Short-\nlist (Balunovi\u00b4 c et al., 2025), IMOAnswerBench (Luong et al., 2025), and PutnamBench (Tsoukalas\net al., 2024).\nFor code, we evaluate DeepSeek-V4 series on LiveCodeBench-v6 and an internal Codeforces\nbenchmark. For Codeforces, we collect 14 Codeforces Division 1 contests comprising 114\nproblems (May 2025 - November 2025). The Elo rating is computed as follows. For each contest,\nwe generate 32 candidate solutions per problem. For each problem independently, we sample\n10 of these solutions without replacement and arrange them in a random order to form the\nsubmission sequence. Each submission is judged against a test suite constructed by domain\nexperts. The score for a solved problem follows the penalty scheme of OpenAI (2025): the model\nreceives the median score of human participants who solved the same problem with the same\nnumber of prior failed attempts. This yields a total contest score for each sampled submission\nsequence, which is then converted into a contest rank and subsequently into an estimated rating\nvia the standard Codeforces rating system. The contest-level expected rating is defined as the\nexpectation of this estimated rating over all possible random selections and orderings of the\n10 submissions per problem. The model\u2019s overall rating is the average of these contest-level\nexpected ratings across all 14 contests.\nFor reasoning and knowledge tasks, we set the temperature to 1.0 and the context window to\n8K, 128K, and 384K tokens for the Non-think, High, and Max modes, respectively. For math tasks\n(e.g., HMMT, IMOAnswerBench, Apex, and HLE), we evaluate using the following template:\n\"{question}\\nPlease reason step by step, and put your final answer within\n36" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Standard Benchmark Evaluation", | |
| "node_id": "0042", | |
| "start_index": 36, | |
| "end_index": 36, | |
| "summary": "The partial document discusses the architecture and optimizations of DSec, a system designed for efficient management of microVMs and sandboxes under massive concurrency. It explains the use of the overlaybd disk format for microVMs, enabling cross-instance sharing of a read-only base layer on 3FS and local copy-on-write layers for writes, supporting efficient versioning and fast resumption. The document highlights density optimizations addressing resource bottlenecks by reducing duplicate page-cache footprints and spinlock contention, thereby improving memory overcommitment and CPU efficiency to increase sandbox packing density. It also details trajectory logging for each sandbox, which records command invocations and results to enable fast client recovery, fine-grained provenance tracking, and deterministic replay of sessions. The latter part introduces the evaluation setup for benchmarking knowledge and reasoning capabilities using various datasets and contests, including MMLU-Pro, GPQA, and Codeforces. It describes the methodology for assessing code performance on Codeforces contests by generating candidate solutions, scoring them based on human participant benchmarks, and computing an Elo rating. The evaluation settings for reasoning and math tasks are also outlined, specifying temperature and context window sizes along with a step-by-step reasoning template for math problem-solving.", | |
| "text": "meanwhile, data blocks are fetched from 3FS upon request. For microVMs, DSec uses the\noverlaybd (Li et al., 2020) disk format: the read-only base layer resides on 3FS for cross-\ninstance sharing, while writes go to a local copy-on-write layer. Such snapshots are chainable,\nfacilitating efficient versioning and millisecond-scale resumption.\nDensity Optimizations Under Massive Concurrency.To accommodate hundreds of thousands\nof sandboxes per cluster, DSec tackles two resource bottlenecks. First, it mitigates duplicate\npage-cache footprints in virtualized environments and applies memory reclamation to enable\nsafe overcommitment. Second, it alleviates spinlock contention in the container runtime and\ntherefore, reduces per-sandbox CPU overhead, significantly increasing per-host packing density.\nTrajectory Logging and Preemption-Safe Resumption.DSec maintains a globally ordered\ntrajectory log for each sandbox, persistently recording every command invocation and its results.\nThe trajectory serves three purposes: (1)client fast-forwarding\u2014 when a training task is\npreempted, sandbox resources are retained nonetheless; upon resumption, DSec replays cached\nresults for previously completed commands, accelerating task recovery whilst also preventing\nerrors from re-execution of non-idempotent operations; (2)fine-grained provenance\u2014 the\norigin and corresponding outcomes of each state change are traceable; (3)deterministic replay\n\u2014 any historical session can be faithfully reproduced from its trajectory.\n5.3. Standard Benchmark Evaluation\n5.3.1. Evaluation Setup\nKnowledge and Reasoning.Knowledge and reasoning datasets include MMLU-Pro (Wang\net al., 2024b), GPQA (Rein et al., 2023), Human Last Exam (Phan et al., 2025), Simple-QA Veri-\nfied (Haas et al., 2025), Chinese-SimpleQA (He et al., 2024), LiveCodeBench-v6 (Jain et al., 2024),\nCodeForces (Internal Benchmark), HMMT 2026 Feb, Apex (Balunovi\u00b4 c et al., 2025), Apex Short-\nlist (Balunovi\u00b4 c et al., 2025), IMOAnswerBench (Luong et al., 2025), and PutnamBench (Tsoukalas\net al., 2024).\nFor code, we evaluate DeepSeek-V4 series on LiveCodeBench-v6 and an internal Codeforces\nbenchmark. For Codeforces, we collect 14 Codeforces Division 1 contests comprising 114\nproblems (May 2025 - November 2025). The Elo rating is computed as follows. For each contest,\nwe generate 32 candidate solutions per problem. For each problem independently, we sample\n10 of these solutions without replacement and arrange them in a random order to form the\nsubmission sequence. Each submission is judged against a test suite constructed by domain\nexperts. The score for a solved problem follows the penalty scheme of OpenAI (2025): the model\nreceives the median score of human participants who solved the same problem with the same\nnumber of prior failed attempts. This yields a total contest score for each sampled submission\nsequence, which is then converted into a contest rank and subsequently into an estimated rating\nvia the standard Codeforces rating system. The contest-level expected rating is defined as the\nexpectation of this estimated rating over all possible random selections and orderings of the\n10 submissions per problem. The model\u2019s overall rating is the average of these contest-level\nexpected ratings across all 14 contests.\nFor reasoning and knowledge tasks, we set the temperature to 1.0 and the context window to\n8K, 128K, and 384K tokens for the Non-think, High, and Max modes, respectively. For math tasks\n(e.g., HMMT, IMOAnswerBench, Apex, and HLE), we evaluate using the following template:\n\"{question}\\nPlease reason step by step, and put your final answer within\n36", | |
| "nodes": [ | |
| { | |
| "title": "Evaluation Setup", | |
| "node_id": "0043", | |
| "start_index": 36, | |
| "end_index": 37, | |
| "summary": "The partial document covers several key topics related to the design, optimization, and evaluation of the DSec system and the DeepSeek-V4 series models. It begins by describing DSec\u2019s storage architecture for microVMs using overlaybd disk format with a read-only base layer on 3FS and a local copy-on-write layer, enabling efficient snapshotting and fast resumption. It then discusses density optimizations under massive concurrency, focusing on reducing memory duplication and spinlock contention to improve resource utilization and sandbox packing density. The document also details DSec\u2019s trajectory logging mechanism, which records all sandbox commands to enable fast client recovery, fine-grained provenance tracking, and deterministic replay of sessions.\n\nThe evaluation section outlines the setup and methodology for benchmarking DeepSeek-V4 models across various knowledge, reasoning, code, and agent tasks. It lists numerous datasets used for knowledge and reasoning evaluation, including math and formal verification tasks with specific prompting templates and agentic formal proof settings. The evaluation also covers long-context performance using 1M-token benchmarks and agent tasks involving code execution and web search, with detailed descriptions of the evaluation frameworks, interaction limits, and context lengths. Finally, preliminary evaluation results comparing DeepSeek-V4 variants to other models are mentioned, highlighting performance metrics on standard benchmarks and agent tasks.", | |
| "text": "meanwhile, data blocks are fetched from 3FS upon request. For microVMs, DSec uses the\noverlaybd (Li et al., 2020) disk format: the read-only base layer resides on 3FS for cross-\ninstance sharing, while writes go to a local copy-on-write layer. Such snapshots are chainable,\nfacilitating efficient versioning and millisecond-scale resumption.\nDensity Optimizations Under Massive Concurrency.To accommodate hundreds of thousands\nof sandboxes per cluster, DSec tackles two resource bottlenecks. First, it mitigates duplicate\npage-cache footprints in virtualized environments and applies memory reclamation to enable\nsafe overcommitment. Second, it alleviates spinlock contention in the container runtime and\ntherefore, reduces per-sandbox CPU overhead, significantly increasing per-host packing density.\nTrajectory Logging and Preemption-Safe Resumption.DSec maintains a globally ordered\ntrajectory log for each sandbox, persistently recording every command invocation and its results.\nThe trajectory serves three purposes: (1)client fast-forwarding\u2014 when a training task is\npreempted, sandbox resources are retained nonetheless; upon resumption, DSec replays cached\nresults for previously completed commands, accelerating task recovery whilst also preventing\nerrors from re-execution of non-idempotent operations; (2)fine-grained provenance\u2014 the\norigin and corresponding outcomes of each state change are traceable; (3)deterministic replay\n\u2014 any historical session can be faithfully reproduced from its trajectory.\n5.3. Standard Benchmark Evaluation\n5.3.1. Evaluation Setup\nKnowledge and Reasoning.Knowledge and reasoning datasets include MMLU-Pro (Wang\net al., 2024b), GPQA (Rein et al., 2023), Human Last Exam (Phan et al., 2025), Simple-QA Veri-\nfied (Haas et al., 2025), Chinese-SimpleQA (He et al., 2024), LiveCodeBench-v6 (Jain et al., 2024),\nCodeForces (Internal Benchmark), HMMT 2026 Feb, Apex (Balunovi\u00b4 c et al., 2025), Apex Short-\nlist (Balunovi\u00b4 c et al., 2025), IMOAnswerBench (Luong et al., 2025), and PutnamBench (Tsoukalas\net al., 2024).\nFor code, we evaluate DeepSeek-V4 series on LiveCodeBench-v6 and an internal Codeforces\nbenchmark. For Codeforces, we collect 14 Codeforces Division 1 contests comprising 114\nproblems (May 2025 - November 2025). The Elo rating is computed as follows. For each contest,\nwe generate 32 candidate solutions per problem. For each problem independently, we sample\n10 of these solutions without replacement and arrange them in a random order to form the\nsubmission sequence. Each submission is judged against a test suite constructed by domain\nexperts. The score for a solved problem follows the penalty scheme of OpenAI (2025): the model\nreceives the median score of human participants who solved the same problem with the same\nnumber of prior failed attempts. This yields a total contest score for each sampled submission\nsequence, which is then converted into a contest rank and subsequently into an estimated rating\nvia the standard Codeforces rating system. The contest-level expected rating is defined as the\nexpectation of this estimated rating over all possible random selections and orderings of the\n10 submissions per problem. The model\u2019s overall rating is the average of these contest-level\nexpected ratings across all 14 contests.\nFor reasoning and knowledge tasks, we set the temperature to 1.0 and the context window to\n8K, 128K, and 384K tokens for the Non-think, High, and Max modes, respectively. For math tasks\n(e.g., HMMT, IMOAnswerBench, Apex, and HLE), we evaluate using the following template:\n\"{question}\\nPlease reason step by step, and put your final answer within\n36\\boxed{}.\" For DeepSeek-V4-Pro-Max on math tasks, we use the following template to elicit\ndeeper reasoning: \"Solve the following problem. The problem may ask you to\nprove a statement, or ask for an answer. If finding an answer is required,\nyou should come up with the answer, and your final solution should also be\na rigorous proof of that answer being valid.\\n\\n{question}\".\nFor formal math tasks, we evaluate in an agentic setting on Lean v4.28.0-rc1 (Moura and\nUllrich, 2021), with access to the Lean compiler and a semantic tactic search engine, running\nup to 500 tool calls with max reasoning effort. In addition, we evaluate a more compute-\nintensive pipeline in which candidate natural-language solutions are first generated and filtered\nby self-verification(Shao et al., 2025), and the retained solutions are then provided as guidance\nto a formal agent for proving the corresponding Lean statement. This design uses informal\nreasoning to improve exploration while preserving strict correctness through formal verification.\nA submission is counted as correct only if the strict verifier Comparator accepts it for both\nsettings.\nWe have left some entries blank for K2.6 and GLM-5.1, as their APIs were too busy to return\nresponses to our queries.\n1M-Token Context.Since DeepSeek-V4 series supports 1M-token contexts, we evaluate model\nperformance in a long context scenario by selecting OpenAI MRCR (OpenAI, 2024b) and\nCorpusQA (Lu et al., 2026) as the benchmarks. We re-evaluate Claude Opus 4.6 and Gemini 3.1\nPro on these tasks with the goal of standardizing the configuration across all models. We did\nnot evaluate GPT-5.4 because its API failed to respond to a large portion of our queries.\nAgent.Agent datasets include Terminal Bench 2.0 (Merrill et al., 2026), SWE-Verified (OpenAI,\n2024e), SWE Multilingual (Yang et al., 2025), SWE-Pro (Deng et al., 2025), BrowseComp (Wei\net al., 2025), the public evaluation set of MCPAtlas (Bandi et al., 2026), GDPval-AA (AA, 2025;\nPatwardhan et al., 2025), and Tool-Decathlon (Li et al., 2025).\nFor code agent tasks (SWE-Verified, Terminal-Bench, SWE-Pro, SWE Multilingual), we eval-\nuate DeepSeek-V4 series using an internally developed evaluation framework. This framework\nprovides a minimal set of tools \u2014 a bash tool and a file-edit tool. The maximum number of\ninteraction steps is set to 500, and the maximum context length is set to 512K tokens. Regarding\nTerminal-Bench 2.0, we acknowledge the environment-related issues noted by GLM-5.1. Never-\ntheless, we report our performance on the original Terminal-Bench 2.0 dataset for consistency.\nOn the Terminal-Bench 2.0 Verified subset, DeepSeek-V4-Pro achieves a score of approximately\n72.0.\nFor search agent tasks (BrowseComp, HLE w/ tool), we also use an in-house harness with\nwebsearch and Python tool, and set maximum interaction steps to 500 and the maximum context\nlength to 512K tokens. For BrowseComp, we use the same discard-all context management\nstrategy as DeepSeek-V3.2 (DeepSeek-AI, 2025).\n5.3.2. Evaluation Results\nThe comparison of DeepSeek-V4-Pro-Max and other closed/open source models is presented in\nTable 6. Also, we evaluate different modes of DeepSeek-V4-Flash and DeepSeek-V4-Pro and\nshow the results in Table 7.\n37" | |
| }, | |
| { | |
| "title": "Evaluation Results", | |
| "node_id": "0044", | |
| "start_index": 37, | |
| "end_index": 41, | |
| "summary": "The partial document details the evaluation methodology and performance results of the DeepSeek-V4 series of large language models across various tasks. It covers the use of specialized prompting templates for deep mathematical reasoning and formal verification using Lean, including a hybrid informal-formal pipeline to enhance proof rigor. The document discusses benchmarking on long-context tasks (up to 1 million tokens) using datasets like MRCR and CorpusQA, and agentic tasks involving code and search agents evaluated with custom frameworks and toolsets. Comparative results highlight DeepSeek-V4-Pro-Max\u2019s state-of-the-art performance among open-source models in knowledge, reasoning, and agentic benchmarks, often surpassing prior open models and approaching or matching proprietary models on many metrics. The text also emphasizes the impact of reasoning effort modes on performance, with higher effort yielding better results. Additionally, it presents DeepSeek-V4\u2019s strong formal math reasoning capabilities under both practical and compute-intensive regimes, outperforming previous systems. The document concludes with real-world task evaluations focusing on Chinese writing, where DeepSeek-V4-Pro demonstrates superior functional and creative writing performance compared to leading external models, though some challenges remain against certain competitors in complex scenarios.", | |
| "text": "\\boxed{}.\" For DeepSeek-V4-Pro-Max on math tasks, we use the following template to elicit\ndeeper reasoning: \"Solve the following problem. The problem may ask you to\nprove a statement, or ask for an answer. If finding an answer is required,\nyou should come up with the answer, and your final solution should also be\na rigorous proof of that answer being valid.\\n\\n{question}\".\nFor formal math tasks, we evaluate in an agentic setting on Lean v4.28.0-rc1 (Moura and\nUllrich, 2021), with access to the Lean compiler and a semantic tactic search engine, running\nup to 500 tool calls with max reasoning effort. In addition, we evaluate a more compute-\nintensive pipeline in which candidate natural-language solutions are first generated and filtered\nby self-verification(Shao et al., 2025), and the retained solutions are then provided as guidance\nto a formal agent for proving the corresponding Lean statement. This design uses informal\nreasoning to improve exploration while preserving strict correctness through formal verification.\nA submission is counted as correct only if the strict verifier Comparator accepts it for both\nsettings.\nWe have left some entries blank for K2.6 and GLM-5.1, as their APIs were too busy to return\nresponses to our queries.\n1M-Token Context.Since DeepSeek-V4 series supports 1M-token contexts, we evaluate model\nperformance in a long context scenario by selecting OpenAI MRCR (OpenAI, 2024b) and\nCorpusQA (Lu et al., 2026) as the benchmarks. We re-evaluate Claude Opus 4.6 and Gemini 3.1\nPro on these tasks with the goal of standardizing the configuration across all models. We did\nnot evaluate GPT-5.4 because its API failed to respond to a large portion of our queries.\nAgent.Agent datasets include Terminal Bench 2.0 (Merrill et al., 2026), SWE-Verified (OpenAI,\n2024e), SWE Multilingual (Yang et al., 2025), SWE-Pro (Deng et al., 2025), BrowseComp (Wei\net al., 2025), the public evaluation set of MCPAtlas (Bandi et al., 2026), GDPval-AA (AA, 2025;\nPatwardhan et al., 2025), and Tool-Decathlon (Li et al., 2025).\nFor code agent tasks (SWE-Verified, Terminal-Bench, SWE-Pro, SWE Multilingual), we eval-\nuate DeepSeek-V4 series using an internally developed evaluation framework. This framework\nprovides a minimal set of tools \u2014 a bash tool and a file-edit tool. The maximum number of\ninteraction steps is set to 500, and the maximum context length is set to 512K tokens. Regarding\nTerminal-Bench 2.0, we acknowledge the environment-related issues noted by GLM-5.1. Never-\ntheless, we report our performance on the original Terminal-Bench 2.0 dataset for consistency.\nOn the Terminal-Bench 2.0 Verified subset, DeepSeek-V4-Pro achieves a score of approximately\n72.0.\nFor search agent tasks (BrowseComp, HLE w/ tool), we also use an in-house harness with\nwebsearch and Python tool, and set maximum interaction steps to 500 and the maximum context\nlength to 512K tokens. For BrowseComp, we use the same discard-all context management\nstrategy as DeepSeek-V3.2 (DeepSeek-AI, 2025).\n5.3.2. Evaluation Results\nThe comparison of DeepSeek-V4-Pro-Max and other closed/open source models is presented in\nTable 6. Also, we evaluate different modes of DeepSeek-V4-Flash and DeepSeek-V4-Pro and\nshow the results in Table 7.\n37Table 6|Comparison between DeepSeek-V4-Pro-Max and closed/open source models. \"Max\",\n\"xHigh\", and \"High\" denote reasoning effort. The best results are highlighted in bold; the\nsecond-best results are underlined.\nBenchmark(Metric)Opus-4.6 GPT-5.4 Gemini-3.1-Pro K2.6 GLM-5.1 DS-V4-Pro\nMax xHigh High Thinking Thinking MaxKnowledge & ReasoningMMLU-Pro(EM) 89.1 87.591.0 87.1 86.0 87.5\nSimpleQA-Verified(Pass@1) 46.2 45.375.6 36.9 38.1 57.9\nChinese-SimpleQA(Pass@1) 76.4 76.885.9 75.9 75.0 84.4\nGPQA Diamond(Pass@1) 91.3 93.0 94.3 90.5 86.2 90.1\nHLE(Pass@1) 40.0 39.844.4 36.4 34.7 37.7\nLiveCodeBench(Pass@1) 88.8 - 91.7 89.6 -93.5\nCodeforces(Rating) - 3168 3052 - -3206\nHMMT 2026 Feb(Pass@1) 96.2 97.794.7 92.7 89.4 95.2\nIMOAnswerBench(Pass@1) 75.391.481.0 86.0 83.8 89.8\nApex(Pass@1) 34.5 54.1 60.9 24.0 11.5 38.3\nApex Shortlist(Pass@1) 85.9 78.1 89.1 75.5 72.490.2LongMRCR 1M(MMR) 92.9- 76.3 - - 83.5\nCorpusQA 1M(ACC) 71.7- 53.8 - - 62.0AgenticTerminal Bench 2.0(Acc) 65.475.168.5 66.7 63.5 67.9\nSWE Verified(Resolved) 80.8- 80.6 80.2 - 80.6\nSWE Pro(Resolved) 57.3 57.7 54.2 58.658.4 55.4\nSWE Multilingual(Resolved) 77.5- - 76.7 73.3 76.2\nBrowseComp(Pass@1) 83.7 82.785.9 83.2 79.3 83.4\nHLE w/ tools(Pass@1) 53.1 52.0 51.6 54.050.4 48.2\nGDPval-AA(Elo) 1619 16741314 1482 1535 1554\nMCPAtlas Public(Pass@1) 73.867.2 69.2 66.6 71.8 73.6\nToolathlon(Pass@1) 47.254.648.8 50.0 40.7 51.8\nKnowledge.In the evaluation of general world knowledge, DeepSeek-V4-Pro-Max, the max-\nimum reasoning effort mode of DeepSeek-V4-Pro, establishes a new state-of-the-art among\nopen-source large language models. As demonstrated by the SimpleQA-Verified, DeepSeek-V4-\nPro-Max significantly outperforms all existing open-source baselines by a margin of 20 absolute\npercentage points. Despite these advances, it currently trails the leading proprietary model,\nGemini-3.1-Pro. In the domain of educational knowledge and reasoning, DeepSeek-V4-Pro-Max\nmarginally outperforms Kimi and GLM across the MMLU-Pro, GPQA, and HLE benchmarks,\nalthough it lags behind leading proprietary models. Broadly, DeepSeek-V4-Pro-Max marks a\nsignificant milestone in enhancing the world knowledge capabilities of open-source models.\nIn addition, a significant performance gap exists between DeepSeek-V4-Flash and DeepSeek-\nV4-Pro on knowledge-based tasks; this is anticipated, as larger parameter counts facilitate\ngreater knowledge retention during pre-training. Notably, both models demonstrate improved\nresults on knowledge benchmarks when allocated higher reasoning effort.\nReasoning.DeepSeek-V4-Pro-Max outperforms all prior open models across reasoning bench-\nmarks, and matches state-of-the-art closed models on many metrics, while the smaller DeepSeek-\nV4-Flash-Max also surpasses the previous best open-source model, K2.6-Thinking, on code\nand math reasoning tasks. Meanwhile, DeepSeek-V4-Pro and DeepSeek-V4-Flash excel in cod-\n38Table 7|Comparison among different sizes and modes of DeepSeek-V4 series. \"Non-Think\",\n\"High\", and \"Max\" denote reasoning effort.\nBenchmark(Metric)DeepSeek-V4-Flash DeepSeek-V4-Pro\nNon-Think High Max Non-Think High MaxKnowledge & ReasoningMMLU-Pro(EM) 83.0 86.4 86.2 82.9 87.1 87.5\nSimpleQA-Verified(Pass@1) 23.1 28.9 34.1 45.0 46.2 57.9\nChinese-SimpleQA(Pass@1) 71.5 73.2 78.9 75.8 77.7 84.4\nGPQA Diamond(Pass@1) 71.2 87.4 88.1 72.9 89.1 90.1\nHLE(Pass@1) 8.1 29.4 34.8 7.7 34.5 37.7\nLiveCodeBench(Pass@1-COT) 55.2 88.4 91.6 56.8 89.8 93.5\nCodeforces(Rating) - 2816 3052 - 2919 3206\nHMMT 2026 Feb(Pass@1) 40.8 91.9 94.8 31.7 94.0 95.2\nIMOAnswerBench(Pass@1) 41.9 85.1 88.4 35.3 88.0 89.8\nApex(Pass@1) 1.0 19.1 33.0 0.4 27.4 38.3\nApex Shortlist(Pass@1) 9.3 72.1 85.7 9.2 85.5 90.2LongMRCR 1M(MMR) 37.5 76.9 78.7 44.7 83.3 83.5\nCorpusQA 1M(ACC) 15.5 59.3 60.5 35.6 56.5 62.0AgenticTerminal Bench 2.0(Acc) 49.1 56.6 56.9 59.1 63.3 67.9\nSWE Verified(Resolved) 73.7 78.6 79.0 73.6 79.4 80.6\nSWE Pro(Resolved) 49.1 52.3 52.6 52.1 54.4 55.4\nSWE Multilingual(Resolved) 69.7 70.2 73.3 69.8 74.1 76.2\nBrowseComp(Pass@1) - 53.5 73.2 - 80.4 83.4\nHLE w/ tools(Pass@1) - 40.3 45.1 - 44.7 48.2\nMCPAtlas Public(Pass@1) 64.0 67.4 69.0 69.4 74.2 73.6\nGDPval-AA(Elo) - - 1395 - - 1554\nToolathlon(Pass@1) 40.7 43.5 47.8 46.3 49.0 51.8\ning competitions. According to our evaluation, their performance is comparable to GPT-5.4,\nmaking this the first time an open model has matched a closed model on this task. On the\nCodeforces leaderboard, DeepSeek-V4-Pro-Max currently ranks 23rd among human candidates.\nDeepSeek-V4 also demonstrates strong performance on formal mathematical task under both\nagentic and compute-intensive settings. Under an agentic setup, it achieves state-of-the-art\nresults, shown in Figure 8, outperforming prior models such as Seed Prover (Chen et al., 2025).\nWith a more compute-intensive pipeline, performance further improves, surpassing systems\nincluding Aristotle(Achim et al., 2025) and matching the best known results under this setting.\nAgent.The DeepSeek-V4 series demonstrates strong agent performance in evaluations. For\ncode agent tasks, DeepSeek-V4-Pro achieves results comparable to K2.6 and GLM-5.1, though\nall these open models still lag behind their closed-source counterparts. DeepSeek-V4-Flash\nunderperforms DeepSeek-V4-Pro on coding tasks, particularly on Terminal Bench 2.0. A similar\ntrend is observed across other agent evaluations. It is worth noting that DeepSeek-V4-Pro\nperforms well on MCPAtlas and Toolathlon\u2014two evaluation test sets that include a wide range\nof tools and MCP services\u2014indicating that our model has excellent generalization capability\nand does not perform well only on internal frameworks.\n1M-Token Context.DeepSeek-V4-Pro outperforms Gemini-3.1-Pro on the MRCR task, which\nmeasures in-context retrieval, but remains behind Claude Opus 4.6. As illustrated in Figure 9,\nretrieval performance remains highly stable within a 128K context window. While a performance\n39Practical Regime\nPutnam-200 Pass@8 with minimal tools\nand bounded sampling.\nSeed-1.5-Prover\n26.50\nGemini-3-Pro\n26.50\nSeed-2.0-Pro\n35.50\nDeepSeek-V4-Flash-Max\n81.00\nFrontier Regime\nPutnam-2025 with hybrid formal-informal\nreasoning and substantial compute scaling.\nAristotle\n100/120\nSeed-1.5-Prover\n110/120\nAxiom\n120/120\nDeepSeek-V4\n120/120\nFigure 8|Formal reasoning under practical and frontier regimes. Left: Putnam-200 Pass@8\nevaluates a fixed random subset of PutnamBench (Tsoukalas et al., 2024) following the setup\nintroduced by Seed-Prover; all models are tested on the same problem set. We follow the Seed-\nProver protocol but replace proprietary search tools with the open-source LeanExplore (Asher,\n2025), yielding a lightweight setting with minimal agent tools and bounded sampling. Right:\nPutnam-2025 probes the frontier of mathematical reasoning in a scaled hybrid formal-informal\nregime, where informal reasoning is combined with formal verification to expose gaps and\nimprove rigor; DeepSeek-V4 reaches a proof-perfect 120/120.\n8k 16k 32k 64k 128k 256k 512k 1024k\nInput Tokens0.00.20.40.60.81.0Average MMR0.90\n0.850.94\n0.90 0.92\n0.82\n0.66\n0.590.91\n0.840.870.850.87\n0.76\n0.60\n0.49MRCR 8-needle\nDeepSeek-V4-Pro-Max\nDeepSeek-V4-Flash-Max\nFigure 9|DeepSeek-V4 series performance on the MRCR task.\ndegradation becomes visible beyond the 128K mark, the model\u2019s retrieval capabilities at 1M\ntokens remain remarkably strong compared to both proprietary and open-source counterparts.\nUnlike MRCR, CorpusQA is similar to real scenarios. The evaluation results also indicate that\nDeepSeek-V4-Pro is better than Gemini-3.1-Pro.\nReasoning Effort.As shown in Table 7, the Max mode, which employs longer contexts and\nreduced length penalties in RL, outperforms the High mode on the most challenging tasks.\nFigure 10 presents a comparison of performance and cost among DeepSeek-V4-Pro, DeepSeek-\nV4-Flash, and DeepSeek-V3.2 on representative reasoning and agentic tasks. By scaling test-time\ncompute, DeepSeek-V4 series achieve substantial improvements over the predecessor. Further-\nmore, on reasoning tasks like HLE, DeepSeek-V4-Pro demonstrates higher token efficiency than\nDeepSeek-V3.2.\n400 20k 40k 60k 80k\nTotal Tokens510152025303540Pass@1 (%)\nNoneThinkSpeciale\nNoneHighMax\nNoneHighMaxHLE\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nDeepSeek-V3.2\n20k 30k 40k 50k\nTotal Tokens3040506070Pass@1 (%)\nNoneThink\nNoneHigh Max\nNoneHighMaxTerminalBench 2.0\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nDeepSeek-V3.2Figure 10|HLE and Terminal Bench 2.0 performance by reasoning effort. \u201cNone\u201d indicates\nNon-think mode, and \u201cSpeciale\u201d indicates DeepSeek-V3.2-Speciale model.\n5.4. Performance on Real-World Tasks\nStandardized benchmarks often struggle to capture the complexities of diverse, real-world\ntasks, creating a gap between test results and actual user experience. To bridge this, we have\ndeveloped proprietary internal metrics that prioritize real-world usage patterns over traditional\nbenchmarks. This approach ensures that our optimizations translate into tangible benefits.\nOur evaluation framework specifically targets the primary use cases of the DeepSeek API and\nChatbot, aligning model performance with practical demands.\n5.4.1. Chinese Writing\nOne of the primary use cases for DeepSeek is Chinese writing. We conducted a rigorous\nevaluation on functional writing and creative writing. Table 12 presents a pairwise comparison\nbetween DeepSeek-V4-Pro and Gemini-3.1-Pro on functional writing tasks. These tasks consist\nof common daily writing queries, where prompts are typically concise and straightforward.\nGemini-3.1-Pro was selected as the baseline, as it stands as the top-performing external model\nfor Chinese writing in our evaluations. The results indicate that DeepSeek-V4-Pro outperforms\nthe baseline with an overall win rate of 62.7% versus 34.1%; this is primarily because Gemini\noccasionally allows its inherent stylistic preferences to override the user\u2019s explicit requirements\nin Chinese writing scenarios.\nTable 13 presents the creative writing comparison, which is evaluated along two axes:\ninstruction following and writing quality. Compared with Gemini-3.1-Pro, DeepSeek-V4-Pro\nachieves a 60.0% win rate in instruction following and 77.5% in writing quality, demonstrating\na marginal improvement in instruction following and a substantial gain in writing quality.\nAlthough DeepSeek-V4-Pro yields superior results in aggregate user case analysis, an evaluation\nrestricted to the most challenging prompts \u2014 specifically those involving high-complexity\nconstraints or multi-turn scenarios \u2014 reveals that Claude Opus 4.5 retains a performance\nadvantage over DeepSeek-V4-Pro. As shown in Table 14, Claude Opus 4.5 achieves a 52.0% win\nrate versus 45.9%.\n41" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Performance on Real-World Tasks", | |
| "node_id": "0045", | |
| "start_index": 41, | |
| "end_index": 41, | |
| "summary": "The partial document discusses the evaluation of DeepSeek models, particularly DeepSeek-V4-Pro, on real-world tasks with a focus on Chinese writing. It highlights the limitations of standardized benchmarks and introduces proprietary internal metrics designed to better reflect practical usage. The evaluation compares DeepSeek-V4-Pro against other leading models like Gemini-3.1-Pro and Claude Opus 4.5 across functional and creative writing tasks. Results show DeepSeek-V4-Pro outperforming Gemini-3.1-Pro in both functional writing and creative writing quality, while Claude Opus 4.5 maintains an edge in handling the most complex and multi-turn prompts. The document also references performance metrics related to reasoning effort and model variants.", | |
| "text": "0 20k 40k 60k 80k\nTotal Tokens510152025303540Pass@1 (%)\nNoneThinkSpeciale\nNoneHighMax\nNoneHighMaxHLE\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nDeepSeek-V3.2\n20k 30k 40k 50k\nTotal Tokens3040506070Pass@1 (%)\nNoneThink\nNoneHigh Max\nNoneHighMaxTerminalBench 2.0\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nDeepSeek-V3.2Figure 10|HLE and Terminal Bench 2.0 performance by reasoning effort. \u201cNone\u201d indicates\nNon-think mode, and \u201cSpeciale\u201d indicates DeepSeek-V3.2-Speciale model.\n5.4. Performance on Real-World Tasks\nStandardized benchmarks often struggle to capture the complexities of diverse, real-world\ntasks, creating a gap between test results and actual user experience. To bridge this, we have\ndeveloped proprietary internal metrics that prioritize real-world usage patterns over traditional\nbenchmarks. This approach ensures that our optimizations translate into tangible benefits.\nOur evaluation framework specifically targets the primary use cases of the DeepSeek API and\nChatbot, aligning model performance with practical demands.\n5.4.1. Chinese Writing\nOne of the primary use cases for DeepSeek is Chinese writing. We conducted a rigorous\nevaluation on functional writing and creative writing. Table 12 presents a pairwise comparison\nbetween DeepSeek-V4-Pro and Gemini-3.1-Pro on functional writing tasks. These tasks consist\nof common daily writing queries, where prompts are typically concise and straightforward.\nGemini-3.1-Pro was selected as the baseline, as it stands as the top-performing external model\nfor Chinese writing in our evaluations. The results indicate that DeepSeek-V4-Pro outperforms\nthe baseline with an overall win rate of 62.7% versus 34.1%; this is primarily because Gemini\noccasionally allows its inherent stylistic preferences to override the user\u2019s explicit requirements\nin Chinese writing scenarios.\nTable 13 presents the creative writing comparison, which is evaluated along two axes:\ninstruction following and writing quality. Compared with Gemini-3.1-Pro, DeepSeek-V4-Pro\nachieves a 60.0% win rate in instruction following and 77.5% in writing quality, demonstrating\na marginal improvement in instruction following and a substantial gain in writing quality.\nAlthough DeepSeek-V4-Pro yields superior results in aggregate user case analysis, an evaluation\nrestricted to the most challenging prompts \u2014 specifically those involving high-complexity\nconstraints or multi-turn scenarios \u2014 reveals that Claude Opus 4.5 retains a performance\nadvantage over DeepSeek-V4-Pro. As shown in Table 14, Claude Opus 4.5 achieves a 52.0% win\nrate versus 45.9%.\n41", | |
| "nodes": [ | |
| { | |
| "title": "Chinese Writing", | |
| "node_id": "0046", | |
| "start_index": 41, | |
| "end_index": 41, | |
| "summary": "The partial document discusses the evaluation of DeepSeek models, particularly DeepSeek-V4-Pro, on real-world tasks with a focus on Chinese writing. It highlights the limitations of standardized benchmarks and introduces proprietary internal metrics designed to better reflect practical usage. The evaluation compares DeepSeek-V4-Pro against other leading models like Gemini-3.1-Pro and Claude Opus 4.5 across functional and creative writing tasks. Results show DeepSeek-V4-Pro outperforming Gemini-3.1-Pro in both functional writing and creative writing quality, while Claude Opus 4.5 maintains an edge in handling the most complex and multi-turn prompts. The document also references performance metrics related to reasoning effort and model variants.", | |
| "text": "0 20k 40k 60k 80k\nTotal Tokens510152025303540Pass@1 (%)\nNoneThinkSpeciale\nNoneHighMax\nNoneHighMaxHLE\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nDeepSeek-V3.2\n20k 30k 40k 50k\nTotal Tokens3040506070Pass@1 (%)\nNoneThink\nNoneHigh Max\nNoneHighMaxTerminalBench 2.0\nDeepSeek-V4-Pro\nDeepSeek-V4-Flash\nDeepSeek-V3.2Figure 10|HLE and Terminal Bench 2.0 performance by reasoning effort. \u201cNone\u201d indicates\nNon-think mode, and \u201cSpeciale\u201d indicates DeepSeek-V3.2-Speciale model.\n5.4. Performance on Real-World Tasks\nStandardized benchmarks often struggle to capture the complexities of diverse, real-world\ntasks, creating a gap between test results and actual user experience. To bridge this, we have\ndeveloped proprietary internal metrics that prioritize real-world usage patterns over traditional\nbenchmarks. This approach ensures that our optimizations translate into tangible benefits.\nOur evaluation framework specifically targets the primary use cases of the DeepSeek API and\nChatbot, aligning model performance with practical demands.\n5.4.1. Chinese Writing\nOne of the primary use cases for DeepSeek is Chinese writing. We conducted a rigorous\nevaluation on functional writing and creative writing. Table 12 presents a pairwise comparison\nbetween DeepSeek-V4-Pro and Gemini-3.1-Pro on functional writing tasks. These tasks consist\nof common daily writing queries, where prompts are typically concise and straightforward.\nGemini-3.1-Pro was selected as the baseline, as it stands as the top-performing external model\nfor Chinese writing in our evaluations. The results indicate that DeepSeek-V4-Pro outperforms\nthe baseline with an overall win rate of 62.7% versus 34.1%; this is primarily because Gemini\noccasionally allows its inherent stylistic preferences to override the user\u2019s explicit requirements\nin Chinese writing scenarios.\nTable 13 presents the creative writing comparison, which is evaluated along two axes:\ninstruction following and writing quality. Compared with Gemini-3.1-Pro, DeepSeek-V4-Pro\nachieves a 60.0% win rate in instruction following and 77.5% in writing quality, demonstrating\na marginal improvement in instruction following and a substantial gain in writing quality.\nAlthough DeepSeek-V4-Pro yields superior results in aggregate user case analysis, an evaluation\nrestricted to the most challenging prompts \u2014 specifically those involving high-complexity\nconstraints or multi-turn scenarios \u2014 reveals that Claude Opus 4.5 retains a performance\nadvantage over DeepSeek-V4-Pro. As shown in Table 14, Claude Opus 4.5 achieves a 52.0% win\nrate versus 45.9%.\n41" | |
| }, | |
| { | |
| "title": "Search", | |
| "node_id": "0047", | |
| "start_index": 42, | |
| "end_index": 42, | |
| "summary": "The partial document discusses two main topics: search capabilities and white-collar task performance of the DeepSeek chatbot. In the search section, it contrasts two search methods used by DeepSeek\u2014Retrieval-Augmented Search (RAG) in \"non-think\" mode and agentic search in \"thinking\" mode. It highlights that DeepSeek-V4-Pro significantly outperforms its predecessor DeepSeek-V3.2, especially in single-value search and planning tasks, while noting some competitive areas for V3.2 in comparison and recommendation tasks. Agentic search is emphasized for its iterative tool invocation, superior accuracy on complex queries, and cost efficiency compared to RAG. The white-collar task section details an evaluation of DeepSeek-V4-Pro-Max on 30 advanced Chinese professional tasks across 13 industries, focusing on high-level cognitive demands like analysis, document generation, and editing. Human evaluators assessed outputs on task completion, instruction adherence, content quality, and formatting. Results show DeepSeek-V4-Pro-Max outperforming a competitor model (Opus-4.6-Max) with a strong non-loss rate and notable strengths in task completion and overall performance across diverse enterprise productivity scenarios.", | |
| "text": "5.4.2. Search\nSearch-augmented question answering is a core capability of the DeepSeek chatbot. On the\nDeepSeek web and app, the \"non-think\" mode employs Retrieval-Augmented Search (RAG),\nwhereas the \"thinking\" mode utilizes agentic search.\nRetrieval Augmented Search.We conducted a pairwise evaluation comparing DeepSeek-V4-\nPro and DeepSeek-V3.2 across both objective and subjective Q&A categories. As presented in\nTable 11, DeepSeek-V4-Pro outperforms DeepSeek-V3.2 by a substantial margin, demonstrating\na consistent advantage across both categories. The most pronounced gains are observed in\nsingle-value search and planning & strategy tasks, suggesting that DeepSeek-V4-Pro excels\nat locating precise factual answers and synthesizing structured plans from retrieved context.\nHowever, DeepSeek-V3.2 remains relatively competitive on comparison and recommendation\ntasks, indicating potential room for improvement for DeepSeek-V4-Pro in scenarios requiring\nbalanced, multi-perspective reasoning over search results.\nAgentic Search.Unlike standard RAG, agentic search empowers the model to iteratively\ninvoke search and fetch tools per query, significantly enhancing overall search performance. For\nthe thinking mode in DeepSeek-Chat, we optimized the agentic search function to maximize\nresponse accuracy within a predefined \"thinking budget\". As shown in Table 9, agentic search\nconsistently outperforms RAG, particularly on complex tasks. Furthermore, its cost remains\nhighly efficient, with agentic search being only marginally more expensive than standard RAG\n(see Table 10).\n5.4.3. White-Collar Task\nTo rigorously evaluate the model\u2019s utility in sophisticated enterprise productivity scenarios, we\nconstructed a comprehensive suite of 30 advanced Chinese professional tasks. These workflows\ndeliberately encompass high-level cognitive demands, including in-depth information analysis,\ncomprehensive document generation, and nuanced document editing, spanning a diverse\nspectrum of 13 critical industries (e.g., finance, education, law, and technology). The evaluation\nwas conducted within an in-house agent harness equipped with basic tools, including Bash and\nweb search.\nGiven the open-ended nature of these tasks, automated metrics usually fall short in capturing\nthe nuances of a high-quality response. Therefore, we conducted human evaluations to compare\nthe performance of DeepSeek-V4-Pro-Max against Opus-4.6-Max. Annotators blindly assessed\nthe model outputs across four dimensions:\n\u2022Task Completion:Whether the core problem was successfully resolved.\n\u2022Instruction Following:Adherence to specific constraints and directives.\n\u2022Content Quality:Factual accuracy, logical coherence, and professional tone.\n\u2022Formatting Aesthetics:Layout readability and visual presentation.\nAs illustrated in Figure 11, DeepSeek-V4-Pro-Max outperforms Opus-4.6-Max on diverse\nChinese white-collar tasks, achieving an impressive non-loss rate of 63%, and demonstrating\nconsistent advantages across analysis, generation, and editing tasks. The detailed dimension\nscores shown in Figure 12 highlight the model\u2019s primary strengths in Task Completion and\n42" | |
| }, | |
| { | |
| "title": "White-Collar Task", | |
| "node_id": "0048", | |
| "start_index": 42, | |
| "end_index": 43, | |
| "summary": "The partial document covers two main sections: Search and White-Collar Task evaluation. In the Search section, it discusses DeepSeek chatbot\u2019s two search modes\u2014Retrieval-Augmented Search (RAG) used in \"non-think\" mode and agentic search employed in \"thinking\" mode. It presents comparative evaluations showing DeepSeek-V4-Pro\u2019s superior performance over DeepSeek-V3.2, especially in precise factual retrieval and planning tasks, while noting some competitive aspects of V3.2 in multi-perspective reasoning. Agentic search is highlighted for its iterative tool invocation, improved accuracy on complex queries, and cost efficiency relative to RAG. The White-Collar Task section details an evaluation of DeepSeek-V4-Pro-Max on 30 advanced Chinese professional tasks across 13 industries, emphasizing high-level cognitive demands like analysis, document generation, and editing. Human evaluators assessed outputs on task completion, instruction adherence, content quality, and formatting aesthetics. Results show DeepSeek-V4-Pro-Max outperforming Opus-4.6-Max overall, particularly in task completion and content quality, with strengths in anticipating user intent and producing coherent long-form content. However, it occasionally falls short in strict instruction following, text summarization, and formatting aesthetics, indicating areas for further improvement. The section also references visual data and example outputs illustrating these findings.", | |
| "text": "5.4.2. Search\nSearch-augmented question answering is a core capability of the DeepSeek chatbot. On the\nDeepSeek web and app, the \"non-think\" mode employs Retrieval-Augmented Search (RAG),\nwhereas the \"thinking\" mode utilizes agentic search.\nRetrieval Augmented Search.We conducted a pairwise evaluation comparing DeepSeek-V4-\nPro and DeepSeek-V3.2 across both objective and subjective Q&A categories. As presented in\nTable 11, DeepSeek-V4-Pro outperforms DeepSeek-V3.2 by a substantial margin, demonstrating\na consistent advantage across both categories. The most pronounced gains are observed in\nsingle-value search and planning & strategy tasks, suggesting that DeepSeek-V4-Pro excels\nat locating precise factual answers and synthesizing structured plans from retrieved context.\nHowever, DeepSeek-V3.2 remains relatively competitive on comparison and recommendation\ntasks, indicating potential room for improvement for DeepSeek-V4-Pro in scenarios requiring\nbalanced, multi-perspective reasoning over search results.\nAgentic Search.Unlike standard RAG, agentic search empowers the model to iteratively\ninvoke search and fetch tools per query, significantly enhancing overall search performance. For\nthe thinking mode in DeepSeek-Chat, we optimized the agentic search function to maximize\nresponse accuracy within a predefined \"thinking budget\". As shown in Table 9, agentic search\nconsistently outperforms RAG, particularly on complex tasks. Furthermore, its cost remains\nhighly efficient, with agentic search being only marginally more expensive than standard RAG\n(see Table 10).\n5.4.3. White-Collar Task\nTo rigorously evaluate the model\u2019s utility in sophisticated enterprise productivity scenarios, we\nconstructed a comprehensive suite of 30 advanced Chinese professional tasks. These workflows\ndeliberately encompass high-level cognitive demands, including in-depth information analysis,\ncomprehensive document generation, and nuanced document editing, spanning a diverse\nspectrum of 13 critical industries (e.g., finance, education, law, and technology). The evaluation\nwas conducted within an in-house agent harness equipped with basic tools, including Bash and\nweb search.\nGiven the open-ended nature of these tasks, automated metrics usually fall short in capturing\nthe nuances of a high-quality response. Therefore, we conducted human evaluations to compare\nthe performance of DeepSeek-V4-Pro-Max against Opus-4.6-Max. Annotators blindly assessed\nthe model outputs across four dimensions:\n\u2022Task Completion:Whether the core problem was successfully resolved.\n\u2022Instruction Following:Adherence to specific constraints and directives.\n\u2022Content Quality:Factual accuracy, logical coherence, and professional tone.\n\u2022Formatting Aesthetics:Layout readability and visual presentation.\nAs illustrated in Figure 11, DeepSeek-V4-Pro-Max outperforms Opus-4.6-Max on diverse\nChinese white-collar tasks, achieving an impressive non-loss rate of 63%, and demonstrating\nconsistent advantages across analysis, generation, and editing tasks. The detailed dimension\nscores shown in Figure 12 highlight the model\u2019s primary strengths in Task Completion and\n42Content Quality. Specifically, DeepSeek-V4-Pro-Max proactively anticipates implicit user in-\ntents by frequently providing supplementary insights and self-verification steps. It also excels\nin long-form generation, delivering in-depth, coherent narratives rather than relying on the\noverly simplistic bullet points frequently produced by Opus-4.6-Max. Additionally, the model\nstrictly conforms to formal professional conventions, such as standardized Chinese hierarchical\nnumbering. However, in terms of Instruction Following, it occasionally overlooks specific\nformatting constraints and slightly trails Opus. Furthermore, the model is less proficient at\ncondensing extensive text inputs into succinct summaries. Finally, its Formatting Aesthetics\nstill have substantial room for improvement regarding the overall visual design of presentation\nslides. Figure 13, 14, and 15 present several test cases; due to the extensive length of certain\noutputs, only partial pages are displayed.\n0% 20% 40% 60% 80% 100%\nProportionanalysis\ngeneration\nediting\noverall55.0% 8.0% 37.0%\n52.0% 10.0% 38.0%\n47.0% 18.0% 35.0%\n53.0% 10.0% 37.0%Win Rate: DeepSeek-V4-Pro-Max vs Opus-4.6-Max\nWin Tie Lose\nFigure 11|Win-rate comparison across analy-\nsis, generation, editing tasks, and the overall\nperformance.\nT ask CompletionInstruction FollowingContent Quality\nFormatting AestheticsOverall707580859095100Score98.32\n87.76\n83.32\n76.6886.5296.68\n88.88\n78.00\n72.6884.06Score: DeepSeek-V4-Pro-Max vs Opus-4.6-Max\nDeepSeek-V4-Pro-Max Opus-4.6-MaxFigure 12|Detailed dimension scores includ-\ning Task Completion, Content Quality, Format-\nting Aesthetics, and Instruction Following.\nFigure 13|Example output of a task which requires drafting a joint marketing proposal for a\npopular bubble tea brand and the Beijing Subway.\n43" | |
| }, | |
| { | |
| "title": "Code Agent", | |
| "node_id": "0049", | |
| "start_index": 44, | |
| "end_index": 44, | |
| "summary": "The partial document discusses the evaluation and capabilities of the DeepSeek-V4-Pro coding agent, which was benchmarked using a curated set of challenging real-world R&D tasks from various technology stacks. It highlights DeepSeek-V4-Pro\u2019s superior performance compared to other models and its positive reception among internal developers, despite some minor issues. The conclusion outlines the architectural innovations of the DeepSeek-V4 series, emphasizing its breakthrough in ultra-long-context processing through a hybrid attention mechanism and infrastructure optimizations. The document also addresses the model\u2019s state-of-the-art performance on knowledge and reasoning benchmarks, its cost-efficiency, and future plans to simplify the architecture, improve training stability, and deepen understanding of key components to enhance large-scale model training.", | |
| "text": "5.4.4. Code Agent\nTo benchmark our coding agent capability, we curate tasks from real internal R&D workloads\nWe collect\u223c200 challenging tasks from 50+ internal engineers, spanning feature development,\nbug fixing, refactoring, and diagnostics across diverse technology stacks including PyTorch,\nCUDA, Rust, and C++. Each task is accompanied by its original repository, the corresponding\nexecution environment, and human-annotated scoring rubrics; after rigorous quality filtering,\n30 tasks are retained as the evaluation set. As shown in Table 8, DeepSeek-V4-Pro significantly\noutperforms Claude Sonnet 4.5 and approaches the level of Claude Opus 4.5.\nTable 8|Comparison on R&D Coding Benchmark (external models included strictly for evalua-\ntion purposes).\nModel Haiku 4.5 Sonnet 4.5 DeepSeek-V4-Pro-Max Opus 4.5Opus 4.5\nThinkingOpus 4.6\nThinking\nPass Rate (%) 13 47 67 70 7380\nIn a survey asking DeepSeek developers and researchers ( \ud835\udc41=85) \u2014 all with experience of\nusing DeepSeek-V4-Pro for agentic coding in their daily work \u2014 whether DeepSeek-V4-Pro is\nready to serve as their default and primary coding model compared to other frontier models, 52%\nsaid yes, 39% leaned toward yes, and fewer than 9% said no. Respondents find DeepSeek-V4-Pro\nto deliver satisfactory results across most tasks, but note trivial mistakes, misinterpretation of\nvague prompts, and occasional over-thinking.\n6. Conclusion, Limitations, and Future Directions\nIn this work, we present a preview version of DeepSeek-V4 series, aiming at next-generation\nlarge language models that break the efficiency barrier of ultra-long-context processing. By com-\nbining a hybrid attention architecture that integrates CSA and HCA, DeepSeek-V4 series achieve\na dramatic leap in long-sequence efficiency. The architectural innovations, together with exten-\nsive infrastructure optimization, enable efficient native support for million-token contexts and\nestablish a necessary foundation for future test-time scaling, long-horizon tasks, and emerging\nparadigms such as online learning. Evaluation results demonstrate that DeepSeek-V4-Pro-Max,\nthe maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open\nmodels. It substantially outperforms prior open-source models on knowledge benchmarks,\nachieves superior reasoning performance close to the frontier proprietary models, and delivers\ncompetitive agent capabilities. Meanwhile, DeepSeek-V4-Flash-Max attains comparable reason-\ning performance to leading closed models while maintaining a highly cost-efficient architecture.\nWe believe DeepSeek-V4 series usher in a new era of million-length contexts for open models\nand pave the way toward better efficiency, scale, and intelligence.\nIn pursuit of extreme long-context efficiency, DeepSeek-V4 series adopted a bold architec-\ntural design. To minimize risk, we retained many preliminarily validated components and\ntricks, which, while effective, made the architecture relatively complex. In future iterations,\nwe will carry out more comprehensive and principled investigations to distill the architecture\ndown to its most essential designs, making it more elegant without sacrificing performance.\nMeanwhile, although Anticipatory Routing and SwiGLU Clamping have been proven effective\nin mitigating training instabilities, their underlying principles remain insufficiently understood.\nWe will actively study foundational problems on training stability and strengthen internal metric\nmonitoring, aiming for a more principled and predictive approach to stable large-scale training.\n44" | |
| } | |
| ] | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Conclusion, Limitations, and Future Directions", | |
| "node_id": "0050", | |
| "start_index": 44, | |
| "end_index": 54, | |
| "summary": "The partial document primarily covers the evaluation and benchmarking of the DeepSeek-V4-Pro coding agent, detailing its performance on a curated set of challenging real-world R&D coding tasks across multiple programming languages and technology stacks. It highlights DeepSeek-V4-Pro's superior pass rates compared to other models and presents user survey feedback on its readiness as a primary coding tool. The document also summarizes the architectural innovations of the DeepSeek-V4 series, emphasizing its hybrid attention design for efficient ultra-long-context processing, enabling support for million-token contexts. It discusses the model's state-of-the-art performance on knowledge and reasoning benchmarks, cost-efficiency, and competitive agent capabilities. Additionally, it outlines current limitations, such as architectural complexity and training stability challenges, and future research directions including model sparsity, low-latency deployment, long-horizon agentic tasks, multimodal integration, and improved data curation. The excerpt concludes with an extensive list of references and a comprehensive author and contributor acknowledgment section.", | |
| "text": "5.4.4. Code Agent\nTo benchmark our coding agent capability, we curate tasks from real internal R&D workloads\nWe collect\u223c200 challenging tasks from 50+ internal engineers, spanning feature development,\nbug fixing, refactoring, and diagnostics across diverse technology stacks including PyTorch,\nCUDA, Rust, and C++. Each task is accompanied by its original repository, the corresponding\nexecution environment, and human-annotated scoring rubrics; after rigorous quality filtering,\n30 tasks are retained as the evaluation set. As shown in Table 8, DeepSeek-V4-Pro significantly\noutperforms Claude Sonnet 4.5 and approaches the level of Claude Opus 4.5.\nTable 8|Comparison on R&D Coding Benchmark (external models included strictly for evalua-\ntion purposes).\nModel Haiku 4.5 Sonnet 4.5 DeepSeek-V4-Pro-Max Opus 4.5Opus 4.5\nThinkingOpus 4.6\nThinking\nPass Rate (%) 13 47 67 70 7380\nIn a survey asking DeepSeek developers and researchers ( \ud835\udc41=85) \u2014 all with experience of\nusing DeepSeek-V4-Pro for agentic coding in their daily work \u2014 whether DeepSeek-V4-Pro is\nready to serve as their default and primary coding model compared to other frontier models, 52%\nsaid yes, 39% leaned toward yes, and fewer than 9% said no. Respondents find DeepSeek-V4-Pro\nto deliver satisfactory results across most tasks, but note trivial mistakes, misinterpretation of\nvague prompts, and occasional over-thinking.\n6. Conclusion, Limitations, and Future Directions\nIn this work, we present a preview version of DeepSeek-V4 series, aiming at next-generation\nlarge language models that break the efficiency barrier of ultra-long-context processing. By com-\nbining a hybrid attention architecture that integrates CSA and HCA, DeepSeek-V4 series achieve\na dramatic leap in long-sequence efficiency. The architectural innovations, together with exten-\nsive infrastructure optimization, enable efficient native support for million-token contexts and\nestablish a necessary foundation for future test-time scaling, long-horizon tasks, and emerging\nparadigms such as online learning. Evaluation results demonstrate that DeepSeek-V4-Pro-Max,\nthe maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open\nmodels. It substantially outperforms prior open-source models on knowledge benchmarks,\nachieves superior reasoning performance close to the frontier proprietary models, and delivers\ncompetitive agent capabilities. Meanwhile, DeepSeek-V4-Flash-Max attains comparable reason-\ning performance to leading closed models while maintaining a highly cost-efficient architecture.\nWe believe DeepSeek-V4 series usher in a new era of million-length contexts for open models\nand pave the way toward better efficiency, scale, and intelligence.\nIn pursuit of extreme long-context efficiency, DeepSeek-V4 series adopted a bold architec-\ntural design. To minimize risk, we retained many preliminarily validated components and\ntricks, which, while effective, made the architecture relatively complex. In future iterations,\nwe will carry out more comprehensive and principled investigations to distill the architecture\ndown to its most essential designs, making it more elegant without sacrificing performance.\nMeanwhile, although Anticipatory Routing and SwiGLU Clamping have been proven effective\nin mitigating training instabilities, their underlying principles remain insufficiently understood.\nWe will actively study foundational problems on training stability and strengthen internal metric\nmonitoring, aiming for a more principled and predictive approach to stable large-scale training.\n44In addition, beyond the MoE and sparse attention architecture, we will also proactively explore\nmodel sparsity along new dimensions \u2014 such as more sparse embedding modules (Cheng\net al., 2026) \u2014 to further improve computational and memory efficiency without compromising\ncapability. We will also continuously investigate low-latency architectures and system tech-\nniques to make long-context deployment and interaction more responsive. Furthermore, we\nrecognize the importance and practical value of long-horizon, multi-round agentic tasks, and\nwill continue to iterate and explore in this direction. We are also working on incorporating\nmultimodal capabilities to our models. Finally, we are committed to developing better data\ncuration and synthesis strategies to consistently enhance model intelligence, robustness, and\npractical usability across an increasingly broad range of scenarios and tasks.\nReferences\nAA. Gdpval-aa leaderboard, 2025. URL https://artificialanalysis.ai/methodolog\ny/intelligence-benchmarking#gdpval-aa.\nT. Achim, A. Best, A. Bietti, K. Der, M. F\u00e9d\u00e9rico, S. Gukov, D. Halpern-Leistner, K. Henningsgard,\nY. Kudryashov, A. Meiburg, et al. Aristotle: Imo-level automated theorem proving. arXiv\npreprint arXiv:2510.01346, 2025.\nA. Agache, M. Brooker, A. Florescu, A. Iordache, A. Liguori, R. Neugebauer, P . Piwonka, and\nD.-M. Popa. Firecracker: lightweight virtualization for serverless applications. In Proceedings\nofthe17th Usenix Conference onNetworked Systems Design and Implementation , NSDI\u201920,\npage 419\u2013434, USA, 2020. USENIX Association. ISBN 9781939133137.\nO. J. Aimuyo, B. Oh, and R. Singh. Flashmoe: Fast distributed moe in a single kernel. Advances\ninNeural Information Processing Systems , 2025. URL https://neurips.cc/virtual/2\n025/poster/119124.\nJ. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebr\u00f3n, and S. Sanghai. Gqa: Training\ngeneralized multi-query transformer models from multi-head checkpoints. arXiv preprint\narXiv:2305.13245, 2023.\nJ. Asher. LeanExplore: A search engine for Lean 4 declarations, 2025. URL https://arxiv.or\ng/abs/2506.11085.\nY. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, Z. Chen, J. Cui,\nH. Ding, M. Dong, A. Du, C. Du, D. Du, Y. Du, Y. Fan, Y. Feng, K. Fu, B. Gao, H. Gao, P . Gao,\nT. Gao, X. Gu, L. Guan, H. Guo, J. Guo, H. Hu, X. Hao, T. He, W. He, W. He, C. Hong, Y. Hu,\nZ. Hu, W. Huang, Z. Huang, Z. Huang, T. Jiang, Z. Jiang, X. Jin, Y. Kang, G. Lai, C. Li, F. Li,\nH. Li, M. Li, W. Li, Y. Li, Y. Li, Z. Li, Z. Li, H. Lin, X. Lin, Z. Lin, C. Liu, C. Liu, H. Liu, J. Liu,\nJ. Liu, L. Liu, S. Liu, T. Y. Liu, T. Liu, W. Liu, Y. Liu, Y. Liu, Y. Liu, Y. Liu, Z. Liu, E. Lu, L. Lu,\nS. Ma, X. Ma, Y. Ma, S. Mao, J. Mei, X. Men, Y. Miao, S. Pan, Y. Peng, R. Qin, B. Qu, Z. Shang,\nL. Shi, S. Shi, F. Song, J. Su, Z. Su, X. Sun, F. Sung, H. Tang, J. Tao, Q. Teng, C. Wang, D. Wang,\nF. Wang, and H. Wang. Kimi K2: open agentic intelligence. CoRR , abs/2507.20534, 2025a.\nURLhttps://doi.org/10.48550/arXiv.2507.20534.\nY. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. Longbench\nv2: Towards deeper understanding and reasoning on realistic long-context multitasks. In\nProceedings ofthe63rd Annual Meeting oftheAssociation forComputational Linguistics\n(Volume 1:Long Papers), pages 3639\u20133664, 2025b.\n45M. Balunovi\u00b4 c, J. Dekoninck, I. Petrov, N. Jovanovi\u00b4 c, and M. Vechev. Matharena: Evaluating llms\non uncontaminated math competitions. Proceedings oftheNeural Information Processing\nSystems Track onDatasets and Benchmark, 2025.\nC. Bandi, B. Hertzberg, G. Boo, T. Polakam, J. Da, S. Hassaan, M. Sharma, A. Park, E. Hernandez,\nD. Rambado, et al. Mcp-atlas: A large-scale benchmark for tool-use competency with real\nmcp servers. arXiv preprint arXiv:2602.00933, 2026.\nF. Bellard. Qemu, a fast and portable dynamic translator. In Proceedings oftheAnnual\nConference onUSENIX Annual Technical Conference , ATEC \u201905, page 41, USA, 2005. USENIX\nAssociation.\nI. Bello, H. Pham, Q. V . Le, M. Norouzi, and S. Bengio. Neural combinatorial optimization with\nreinforcement learning, 2017. URLhttps://openreview.net/forum?id=rJY3vK9eg.\nJ. Chen, W. Chen, J. Du, J. Hu, Z. Jiang, A. Jie, X. Jin, X. Jin, C. Li, W. Shi, Z. Wang, M. Wang,\nC. Wei, S. Wei, H. Xin, F. Yang, W. Gao, Z. Yuan, T. Zhan, Z. Zheng, T. Zhou, and T. H.\nZhu. Seed-prover 1.5: Mastering undergraduate-level theorem proving via learning from\nexperience, 2025. URLhttps://arxiv.org/abs/2512.17260.\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P . de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda,\nN. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P . Mishkin,\nB. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P . Tillet,\nF. P . Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss,\nA. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse,\nA. N. Carr, J. Leike, J. Achiam, V . Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage,\nM. Murati, K. Mayer, P . Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and\nW. Zaremba. Evaluating large language models trained on code. CoRR , abs/2107.03374, 2021.\nURLhttps://arxiv.org/abs/2107.03374.\nT. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze,\nC. Guestrin, and A. Krishnamurthy. TVM: An automated End-to-End optimizing com-\npiler for deep learning. In 13th USENIX Symposium onOperating Systems Design and\nImplementation (OSDI 18), pages 578\u2013594, Carlsbad, CA, Oct. 2018. USENIX Association.\nISBN 978-1-939133-08-3. URL https://www.usenix.org/conference/osdi18/prese\nntation/chen.\nA. Cheng, A. Jacovi, A. Globerson, B. Golan, C. Kwong, C. Alberti, C. Tao, E. Ben-David, G. S.\nTomar, L. Haas, et al. The facts leaderboard: A comprehensive benchmark for large language\nmodel factuality. arXiv preprint arXiv:2512.10791, 2025.\nX. Cheng, W. Zeng, D. Dai, Q. Chen, B. Wang, Z. Xie, K. Huang, X. Yu, Z. Hao, Y. Li, H. Zhang,\nH. Zhang, D. Zhao, and W. Liang. Conditional memory via scalable lookup: A new axis of\nsparsity for large language models. CoRR , abs/2601.07372, 2026. doi: 10.48550/ARXIV.2601.\n07372. URLhttps://doi.org/10.48550/arXiv.2601.07372.\nK. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek,\nJ. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint\narXiv:2110.14168, 2021.\nD. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K.\nLi, P . Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. Deepseekmoe: Towards ultimate expert\nspecialization in mixture-of-experts language models. CoRR , abs/2401.06066, 2024. URL\nhttps://doi.org/10.48550/arXiv.2401.06066.\n46T. Dao, D. Haziza, F. Massa, and G. Sizov. Flash-decoding for long-context inference, 2023. URL\nhttps://pytorch.org/blog/flash-decoding/.\nL. De Moura and N. Bj\u00f8rner. Z3: an efficient smt solver. In Proceedings oftheTheory\nand Practice ofSoftware, 14th International Conference onTools and Algorithms forthe\nConstruction and Analysis ofSystems , TACAS\u201908/ETAPS\u201908, page 337\u2013340, Berlin, Heidel-\nberg, 2008. Springer-Verlag. ISBN 3540787992.\nDeepSeek-AI. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelli-\ngence. CoRR , abs/2406.11931, 2024. URL https://doi.org/10.48550/arXiv.2406.11\n931.\nDeepSeek-AI. Deepseek-v3 technical report. CoRR , abs/2412.19437, 2024. URL https://doi.\norg/10.48550/arXiv.2412.19437.\nDeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language\nmodel. CoRR , abs/2405.04434, 2024. URL https://doi.org/10.48550/arXiv.2405.04\n434.\nDeepSeek-AI. Fire-flyer file system, 2025. URLhttps://github.com/deepseek-ai/3FS.\nDeepSeek-AI. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nat. ,\n645(8081):633\u2013638, 2025. URLhttps://doi.org/10.1038/s41586-025-09422-z.\nDeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URL\nhttps://arxiv.org/abs/2512.02556.\nX. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath,\nM. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V . Bharadwaj, J. Holm, R. Aluri, C. B. C.\nZhang, N. Jacobson, B. Liu, and B. Kenstler. Swe-bench pro: Can ai agents solve long-horizon\nsoftware engineering tasks?, 2025. URLhttps://arxiv.org/abs/2509.16941.\nH. Ding, Z. Wang, G. Paolini, V . Kumar, A. Deoras, D. Roth, and S. Soatto. Fewer truncations\nimprove language modeling. arXiv preprint arXiv:2404.10830, 2024.\nX. Dong, Y. Fu, S. Diao, W. Byeon, Z. CHEN, A. S. Mahabaleshwarkar, S.-Y. Liu, M. V . keirsbilck,\nM.-H. Chen, Y. Suhara, Y. C. Lin, J. Kautz, and P . Molchanov. Hymba: A hybrid-head archi-\ntecture for small language models. In The Thirteenth International Conference onLearning\nRepresentations, 2025. URLhttps://openreview.net/forum?id=A1ztozypga.\nX. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, et al. Supergpqa:\nScaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739 , 2025.\nD. Dua, Y. Wang, P . Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading compre-\nhension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and\nT. Solorio, editors, Proceedings ofthe2019 Conference oftheNorth American Chapter ofthe\nAssociation forComputational Linguistics: Human Language Technologies, NAACL-HLT\n2019, Minneapolis, MN, USA, June 2-7,2019, Volume 1(Long and Short Papers) , pages 2368\u2013\n2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL\nhttps://doi.org/10.18653/v1/n19-1246.\nX. Gao, M. Dong, X. Miao, W. Du, C. Yu, and H. Chen. Erofs: a compression-friendly readonly\nfile system for resource-scarce devices. In Proceedings ofthe2019 USENIX Conference on\nUsenix Annual Technical Conference , USENIX ATC \u201919, page 149\u2013162, USA, 2019. USENIX\nAssociation. ISBN 9781939133038.\n47A. P . Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao,\nX. Du, M. R. G. Madani, C. Barale, R. McHardy, J. Harris, J. Kaddour, E. van Krieken, and\nP . Minervini. Are we done with mmlu? CoRR , abs/2406.04127, 2024. URL https://doi.or\ng/10.48550/arXiv.2406.04127.\nF. Gloeckle, B. Y. Idrissi, B. Rozi\u00e8re, D. Lopez-Paz, and G. Synnaeve. Better & faster large\nlanguage models via multi-token prediction. In Forty-first International Conference on\nMachine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL\nhttps://openreview.net/forum?id=pEWAcejiU2.\nY. Gu, L. Dong, F. Wei, and M. Huang. Minillm: Knowledge distillation of large language\nmodels. In The Twelfth International Conference onLearning Representations, 2024.\nL. Haas, G. Yona, G. D\u2019Antonio, S. Goldshtein, and D. Das. Simpleqa verified: A reliable\nfactuality benchmark to measure parametric knowledge. arXiv preprint arXiv:2509.07968 ,\n2025.\nY. He, S. Li, J. Liu, Y. Tan, W. Wang, H. Huang, X. Bu, H. Guo, C. Hu, B. Zheng, et al. Chi-\nnese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint\narXiv:2411.07140, 2024.\nD. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring\nmassive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Mea-\nsuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 ,\n2021.\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A\nmulti-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint\narXiv:2305.08322, 2023.\nD. Hupkes and N. Bogoychev. Multiloko: a multilingual local knowledge benchmark for llms\nspanning 31 languages. CoRR , abs/2504.10356, 2025. doi: 10.48550/ARXIV.2504.10356. URL\nhttps://doi.org/10.48550/arXiv.2504.10356.\nB. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko.\nQuantization and training of neural networks for efficient integer-arithmetic-only inference.\nInProceedings oftheIEEE Conference onComputer Vision and Pattern Recognition (CVPR) ,\nJune 2018.\nN. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica.\nLivecodebench: Holistic and contamination free evaluation of large language models for code.\narXiv preprint arXiv:2403.07974, 2024.\nK. Jordan, Y. Jin, V . Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer\nfor hidden layers in neural networks. Cited on, page 10, 2024.\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised chal-\nlenge dataset for reading comprehension. In R. Barzilay and M.-Y. Kan, editors, Proceedings of\nthe55th Annual Meeting oftheAssociation forComputational Linguistics (Volume 1:Long\nPapers) , pages 1601\u20131611, Vancouver, Canada, July 2017. Association for Computational\nLinguistics. doi: 10.18653/v1/P17-1147. URLhttps://aclanthology.org/P17-1147.\n48H. Li, Y. Yuan, R. Du, K. Ma, L. Liu, and W. Hsu. DADI: Block-Level image service for agile\nand elastic application deployment. In 2020 USENIX Annual Technical Conference (USENIX\nATC 20), pages 727\u2013740. USENIX Association, July 2020. ISBN 978-1-939133-14-4. URL\nhttps://www.usenix.org/conference/atc20/presentation/li-huiba.\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measur-\ning massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212 ,\n2023.\nJ. Li, W. Zhao, J. Zhao, W. Zeng, H. Wu, X. Wang, R. Ge, Y. Cao, Y. Huang, W. Liu, et al. The\ntool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task\nexecution. arXiv preprint arXiv:2510.25726, 2025.\nY. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: speculative sampling requires rethinking\nfeature uncertainty. In Forty-first International Conference onMachine Learning, ICML 2024,\nVienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net\n/forum?id=1NdN7eXyb4.\nJ. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng,\nY. Liu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang,\nX. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang. Muon is scalable for LLM training. CoRR ,\nabs/2502.16982, 2025. URLhttps://doi.org/10.48550/arXiv.2502.16982.\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint\narXiv:1711.05101, 2017.\nK. Lu and T. M. Lab. On-policy distillation. Thinking Machines Lab: Connectionism , 2025. doi:\n10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.\nZ. Lu, C. Li, Y. Shi, W. Shen, M. Yan, and F. Huang. Corpusqa: A 10 million token benchmark\nfor corpus-level analysis and reasoning. arXiv preprint arXiv:2601.14952, 2026.\nT. Luong, D. Hwang, H. H. Nguyen, G. Ghiasi, Y. Chervonyi, I. Seo, J. Kim, G. Bingham,\nJ. Lee, S. Mishra, A. Zhai, C. H. Hu, H. Michalewski, J. Kim, J. Ahn, J. Bae, X. Song, T. H.\nTrinh, Q. V . Le, and J. Jung. Towards robust mathematical reasoning. In Proceedings of\nthe2025 Conference onEmpirical Methods inNatural Language Processing , 2025. URL\nhttps://aclanthology.org/2025.emnlp-main.1794/.\nM. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K.\nBuchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command\nline interfaces. arXiv preprint arXiv:2601.11868, 2026.\nMiniMax. Meet minimax-m2, 2025. URLhttps://github.com/MiniMax-AI/MiniMax-M2.\nL. d. Moura and S. Ullrich. The lean 4 theorem prover and programming language. In\nInternational Conference onAutomated Deduction, pages 625\u2013635. Springer, 2021.\nY. Nesterov. A method of solving a convex programming problem with convergence rate \ud835\udc42(1/\ud835\udc582).\nSoviet Mathematics Doklady, 27:372\u2013376, 1983.\nNVIDIA Corporation. cublas documentation, 2024. URL https://docs.nvidia.com/cuda\n/cublas/. Version 12.4. Accessed: 2024-09-16.\nOpenAI. Multilingual massive multitask language understanding (mmmlu), 2024a. URL\nhttps://huggingface.co/datasets/openai/MMMLU.\n49OpenAI. Openai mrcr: Long context multiple needle in a haystack benchmark, 2024b. URL\nhttps://huggingface.co/datasets/openai/mrcr.\nOpenAI. Learning to reason with llms, 2024c. URL https://openai.com/index/learnin\ng-to-reason-with-llms.\nOpenAI. Introducing SimpleQA, 2024d. URL https://openai.com/index/introducing\n-simpleqa/.\nOpenAI. Introducing SWE-bench verified we\u2019re releasing a human-validated subset of swe-\nbench that more, 2024e. URL https://openai.com/index/introducing-swe-bench-v\nerified/.\nOpenAI. gpt-oss-120b & gpt-oss-20b model card. CoRR , abs/2508.10925, 2025. doi: 10.48550/A\nRXIV.2508.10925. URLhttps://doi.org/10.48550/arXiv.2508.10925.\nM. Osama, D. Merrill, C. Cecka, M. Garland, and J. D. Owens. Stream-k: Work-centric parallel\ndecomposition for dense matrix-matrix multiplication on the gpu. In Proceedings ofthe28th\nACM SIGPLAN Annual Symposium onPrinciples and Practice ofParallel Programming ,\npages 429\u2013431, 2023.\nT. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. P . Fishman, M. Aljubeh,\nP . Thacker, L. Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world\neconomically valuable tasks. arXiv preprint arXiv:2510.04374, 2025.\nL. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al.\nHumanity\u2019s last exam. arXiv preprint arXiv:2501.14249, 2025.\nW. Qi, Y. Yan, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou. Prophetnet: Predicting\nfuture n-gram for sequence-to-sequence pre-training. In T. Cohn, Y. He, and Y. Liu, edi-\ntors, Findings oftheAssociation forComputational Linguistics: EMNLP 2020, Online Event,\n16-20 November 2020 , volume EMNLP 2020 of Findings ofACL , pages 2401\u20132410. Associa-\ntion for Computational Linguistics, 2020. URL https://doi.org/10.18653/v1/2020.f\nindings-emnlp.217.\nQwen. Qwen3 technical report. CoRR , abs/2505.09388, 2025. doi: 10.48550/ARXIV.2505.09388.\nURLhttps://doi.org/10.48550/arXiv.2505.09388.\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training tril-\nlion parameter models. In SC20: International Conference forHigh Performance Computing,\nNetworking, Storage and Analysis, pages 1\u201316. IEEE, 2020.\nJ. K. Reed, Z. DeVito, H. He, A. Ussery, and J. Ansel. Torch.fx: Practical program capture and\ntransformation for deep learning in python, 2022. URL https://arxiv.org/abs/2112.0\n8429.\nD. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.\nGPQA: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 , 2023.\nG. T. M. Riviere, S. Pathak, P . G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard,\nB. Shahriari, A. Ram\u2019e, J. Ferret, P . Liu, P . D. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar,\nC. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P . Sta\u00b4 nczyk, S. Girgin, N. Momchev, M. Hoff-\nman, S. Thakoor, J.-B. Grill, B. Neyshabur, A. Walton, A. Severyn, A. Parrish, A. Ahmad,\nA. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson,\n50B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. A. Welty, C. A. Choquette-\nChoo, D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozi\u2019nska, D. Herbison, E. Bandy,\nE. Wang, E. Noland, E. Moreira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron,\nG. Martins, H. Hashemi, H. Klimczak-Pluci\u2019nska, H. Batra, H. Dhand, I. Nardini, J. Mein,\nJ. Zhou, J. Svensson, J. Stanway, J. Chan, J. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernan-\ndez, J. R. van Amersfoort, J. Gordon, J. Lipschultz, J. Newlan, J. Ji, K. Mohamed, K. Badola,\nK. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui,\nL. Sifre, L. Heuermann, L. cia Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. L. B.\nMartins, M. Reid, M. Singh, M. Iverson, M. Gorner, M. Velloso, M. Wirth, M. Davidow,\nM. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng,\nM. Park, M. Rahman, M. Khatwani, N. Dao, N. shad Bardoliwalla, N. Devanathan, N. Dumai,\nN. Chauhan, O. Wahltinez, P . Botarda, P . Barnes, P . Barham, P . Michel, P . chong Jin, P . Georgiev,\nP . Culliton, P . Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins,\nS. Saadat, S. M. M. Carthy, S. Perrin, S. M. R. Arnold, S. bastian Krause, S. Dai, S. Garg,\nS. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisk\u00fd, T. Doshi,\nV . Jain, V . Yadav, V . Meshram, V . Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon,\nX. Xu, Z. Shen, Z. Gong, Z. Wei, V . Cotruta, P . Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin,\nE. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov,\nO. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud,\nN. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev. Gemma 2: Improving open\nlanguage models at a practical size. arXiv preprint arXiv:2408.00118, 2024.\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. In\nM. Ranzato, A. Beygelzimer, Y. N. Dauphin, P . Liang, and J. W. Vaughan, editors, Advances\ninNeural Information Processing Systems 34:Annual Conference onNeural Information\nProcessing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 17555\u201317566,\n2021. URL https://proceedings.neurips.cc/paper/2021/hash/92bf5e6240737\ne0326ea59846a83e076-Abstract.html.\nB. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea,\nE. Dellinger, K. Denolf, S. Dusan, V . Elango, M. Golub, A. Heinecke, P . James-Roxby, D. Jani,\nG. Kolhe, M. Langhammer, A. Li, L. Melnick, M. Mesmakhosroshahi, A. Rodriguez, M. Schulte,\nR. Shafipour, L. Shao, M. Siu, P . Dubey, P . Micikevicius, M. Naumov, C. Verrilli, R. Wittig,\nD. Burger, and E. Chung. Microscaling data formats for deep learning, 2023.\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd\nschema challenge at scale, 2019.\nZ. Shao, Y. Luo, C. Lu, Z. Z. Ren, J. Hu, T. Ye, Z. Gou, S. Ma, and X. Zhang. Deepseekmath-v2:\nTowards self-verifiable mathematical reasoning, 2025. URL https://arxiv.org/abs/25\n11.22570.\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR , abs/1911.02150,\n2019. URLhttp://arxiv.org/abs/1911.02150.\nN. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.\nF. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder,\nD. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners.\nInThe Eleventh International Conference onLearning Representations, ICLR 2023, Kigali,\nRwanda, May 1-5,2023 . OpenReview.net, 2023. URL https://openreview.net/forum?i\nd=fR3wGCk-IXp.\n51J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary\nposition embedding. Neurocomputing, 568:127063, 2024.\nM. Suzgun, N. Scales, N. Sch\u00e4rli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V . Le,\nE. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve\nthem. arXiv preprint arXiv:2210.09261, 2022.\nG. Tsoukalas, J. Lee, J. Jennings, J. Xin, M. Ding, M. Jennings, A. Thakur, and S. Chaudhuri.\nPutnambench: Evaluating neural theorem-provers on the putnam mathematical competition,\n2024. URLhttps://arxiv.org/abs/2407.11214.\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, \u0141. Kaiser, and I. Polo-\nsukhin. Attention is all you need. Advances inneural information processing systems , 30,\n2017.\nL. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai. Auxiliary-loss-free load balancing strategy for\nmixture-of-experts. CoRR , abs/2408.15664, 2024a. URL https://doi.org/10.48550/arX\niv.2408.15664.\nL. Wang, Y. Cheng, Y. Shi, Z. Mo, Z. Tang, W. Xie, T. Wu, L. Ma, Y. Xia, J. Xue, et al. Tilelang:\nBridge programmability and performance in modern neural kernels. In The Fourteenth\nInternational Conference onLearning Representations, 2026.\nY. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li,\nM. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and\nchallenging multi-task language understanding benchmark. CoRR , abs/2406.01574, 2024b.\nURLhttps://doi.org/10.48550/arXiv.2406.01574.\nJ. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus,\nand A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv\npreprint arXiv:2504.12516, 2025.\nT. Wei, J. Luan, W. Liu, S. Dong, and B. Wang. Cmath: Can your language model pass chinese\nelementary school math test?, 2023.\nG. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with\nattention sinks. In The Twelfth International Conference onLearning Representations, ICLR\n2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview\n.net/forum?id=NG7sS51zVF.\nZ. Xie, Y. Wei, H. Cao, C. Zhao, C. Deng, J. Li, D. Dai, H. Gao, J. Chang, K. Yu, L. Zhao, S. Zhou,\nZ. Xu, Z. Zhang, W. Zeng, S. Hu, Y. Wang, J. Yuan, L. Wang, and W. Liang. mhc: Manifold-\nconstrained hyper-connections, 2026. URLhttps://arxiv.org/abs/2512.24880.\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu,\nB. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou,\nS. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chi-\nnese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors,\nProceedings ofthe28th International Conference onComputational Linguistics, COLING\n2020, Barcelona, Spain (Online), December 8-13, 2020 , pages 4762\u20134772. International Com-\nmittee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL\nhttps://doi.org/10.18653/v1/2020.coling-main.419.\n52J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt,\nand D. Yang. Swe-smith: Scaling data for software engineering agents, 2025. URL https:\n//arxiv.org/abs/2504.21798.\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish\nyour sentence? In A. Korhonen, D. R. Traum, and L. M\u00e0rquez, editors, Proceedings ofthe57th\nConference oftheAssociation forComputational Linguistics, ACL 2019, Florence, Italy, July\n28-August 2,2019, Volume 1:Long Papers , pages 4791\u20134800. Association for Computational\nLinguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1\n9-1472.\nC. Zhang, K. Du, S. Liu, W. Kwon, X. Mo, Y. Wang, X. Liu, K. You, Z. Li, M. Long,\nJ. Zhai, J. Gonzalez, and I. Stoica. Jenga: Effective memory management for serving llm\nwith heterogeneity. In Proceedings oftheACM SIGOPS 31st Symposium onOperating\nSystems Principles , SOSP \u201925, page 446\u2013461, New York, NY, USA, 2025a. Association for\nComputing Machinery. ISBN 9798400718700. doi: 10.1145/3731569.3764823. URL\nhttps://doi.org/10.1145/3731569.3764823.\nS. Zhang, N. Zheng, H. Lin, Z. Jiang, W. Bao, C. Jiang, Q. Hou, W. Cui, S. Zheng, L.-W. Chang,\nQ. Chen, and X. Liu. Comet: Fine-grained computation-communication overlapping for\nmixture-of-experts. 2025b. URLhttps://arxiv.org/abs/2502.19811.\nC. Zhao, L. Zhao, J. Li, Z. Xu, and C. Xu. Deepgemm: clean and efficient fp8 gemm kernels with\nfine-grained scaling.https://github.com/deepseek-ai/DeepGEMM, 2025.\nW. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A\nhuman-centric benchmark for evaluating foundation models. CoRR , abs/2304.06364, 2023.\ndoi: 10.48550/arXiv.2304.06364. URLhttps://doi.org/10.48550/arXiv.2304.06364.\nD. Zhu, H. Huang, Z. Huang, Y. Zeng, Y. Mao, B. Wu, Q. Min, and X. Zhou. Hyper-\nconnections. In The Thirteenth International Conference onLearning Representations, ICLR\n2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL https://openreview.net\n/forum?id=9FqARW7dwB.\nX. Zhu, D. Cheng, H. Li, K. Zhang, E. Hua, X. Lv, N. Ding, Z. Lin, Z. Zheng, and B. Zhou. How\nto synthesize text data without model collapse? arXiv preprint arXiv:2412.14689, 2024.\nT. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul,\nS. Brunner, C. Gong, J. Hoang, A. R. Zebaze, X. Hong, W. Li, J. Kaddour, M. Xu, Z. Zhang,\nP . Yadav, and et al. Bigcodebench: Benchmarking code generation with diverse function\ncalls and complex instructions. In The Thirteenth International Conference onLearning\nRepresentations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL http\ns://openreview.net/forum?id=YrycTjllL0.\n53Appendix\nA. Author List and Acknowledgment\nA.1. Author List\nAuthors are listed alphabetically by their first name. Names marked with * denote individuals\nwho have departed from our team.\nResearch & Engineering:Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang*, Bingzheng Xu,\nBochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi\nDeng, Chenhao Xu, Chenze Shao, Chong Ruan*, Conner Sun, Damai Dai, Daya Guo*, Dejian\nYang, Deli Chen, Donghao Li, Erhang Li, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong\nDai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang,\nHanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei*, Haotian Yuan,\nHaowei Zhang*, Haowen Luo, Haoyu Chen, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi\nCao, Huazuo Gao, Hui Qu, Hui Zeng, J. Yang, J.Q. Zhu, Jia Yu, Jialiang Huang, Jiasheng Ye,\nJiashi Li, Jiaxin Xu, Jiewen Hu, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang\nYuan, Jingyuan Cheng, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran*, Junguang Jiang, Junjie Qiu,\nJunlong Li*, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang*,\nKuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao\nLuo, Linwang Ma, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, M.S. Di, M.Y Xu,\nMax Mei, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang,\nPeixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du,\nQiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan,\nRunji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S.H. Liu,\nShanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoyuan Chen,\nShengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng\nZhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye,\nTianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W. Zhang, Wangding\nZeng, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun\nGao, Wenkai Yang, Wenlve Huang, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen\nWang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie,\nXin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang,\nXu Chen, Xuanyu Wang, Xuecheng Su, Xuheng Lin, Xuwei Fu, Y.C. Yan, Y.Q. Wang*, Y.W. Ma,\nYanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Zhao,\nYaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang\nXiong, Ying He, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang,\nYixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo,\nYongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuhan\nWu, Yuhao Meng, Yuheng Zou, YuKun Li, Yunfan Xiong, Yupeng Chen, Yuqian Cao, Yuqian\nWang, Yushun Zhang, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan\nZhou, Yuyang Zhou, Yuzhen Huang, Z.F. Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zhangli\nSha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma,\nZhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhuoshu\nLi, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang*, Ziwei\nXie, Ziyi Gao, Zizheng Pan, Zongqing Yao.\nBusiness & Compliance:Chenchen Ling, Chengyu Hou, Dongjie Ji, Fang Wei, Hengqing Zhang,\nJia Luo, Jia Song, Jialu Cai, Jian Liang, Jiangting Zhou, Jieyu Yang, Jin Chen, Jingzi Zhou, Junmin\nZheng, Leyi Xia, Linyan Zhu, Miaojun Wang, Mingming Li, Minmin Han, Ning Wang, Panpan\n54" | |
| }, | |
| { | |
| "title": "Author List and Acknowledgment", | |
| "node_id": "0051", | |
| "start_index": 54, | |
| "end_index": 54, | |
| "summary": "The partial document is an appendix section that primarily provides a detailed author list and acknowledgments. It lists contributors alphabetically by their first names, distinguishing those who have left the team with an asterisk. The authors are categorized into groups such as Research & Engineering and Business & Compliance, highlighting the extensive team involved in the project.", | |
| "text": "Appendix\nA. Author List and Acknowledgment\nA.1. Author List\nAuthors are listed alphabetically by their first name. Names marked with * denote individuals\nwho have departed from our team.\nResearch & Engineering:Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang*, Bingzheng Xu,\nBochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi\nDeng, Chenhao Xu, Chenze Shao, Chong Ruan*, Conner Sun, Damai Dai, Daya Guo*, Dejian\nYang, Deli Chen, Donghao Li, Erhang Li, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong\nDai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang,\nHanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei*, Haotian Yuan,\nHaowei Zhang*, Haowen Luo, Haoyu Chen, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi\nCao, Huazuo Gao, Hui Qu, Hui Zeng, J. Yang, J.Q. Zhu, Jia Yu, Jialiang Huang, Jiasheng Ye,\nJiashi Li, Jiaxin Xu, Jiewen Hu, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang\nYuan, Jingyuan Cheng, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran*, Junguang Jiang, Junjie Qiu,\nJunlong Li*, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang*,\nKuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao\nLuo, Linwang Ma, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, M.S. Di, M.Y Xu,\nMax Mei, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang,\nPeixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du,\nQiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan,\nRunji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S.H. Liu,\nShanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoyuan Chen,\nShengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng\nZhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye,\nTianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W. Zhang, Wangding\nZeng, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun\nGao, Wenkai Yang, Wenlve Huang, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen\nWang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie,\nXin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang,\nXu Chen, Xuanyu Wang, Xuecheng Su, Xuheng Lin, Xuwei Fu, Y.C. Yan, Y.Q. Wang*, Y.W. Ma,\nYanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Zhao,\nYaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang\nXiong, Ying He, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang,\nYixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo,\nYongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuhan\nWu, Yuhao Meng, Yuheng Zou, YuKun Li, Yunfan Xiong, Yupeng Chen, Yuqian Cao, Yuqian\nWang, Yushun Zhang, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan\nZhou, Yuyang Zhou, Yuzhen Huang, Z.F. Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zhangli\nSha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma,\nZhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhuoshu\nLi, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang*, Ziwei\nXie, Ziyi Gao, Zizheng Pan, Zongqing Yao.\nBusiness & Compliance:Chenchen Ling, Chengyu Hou, Dongjie Ji, Fang Wei, Hengqing Zhang,\nJia Luo, Jia Song, Jialu Cai, Jian Liang, Jiangting Zhou, Jieyu Yang, Jin Chen, Jingzi Zhou, Junmin\nZheng, Leyi Xia, Linyan Zhu, Miaojun Wang, Mingming Li, Minmin Han, Ning Wang, Panpan\n54", | |
| "nodes": [ | |
| { | |
| "title": "Author List", | |
| "node_id": "0052", | |
| "start_index": 54, | |
| "end_index": 55, | |
| "summary": "The partial document primarily includes an appendix section detailing the author list and acknowledgments, followed by evaluation details of the DeepSeek-V4 series models. The author list is extensive, organized alphabetically by first name, and distinguishes current team members from those who have departed. The acknowledgment section thanks specific individuals for their contributions to model testing and feedback. The evaluation details present comparative performance data between Agentic Search and Retrieval Augmented Search methods for DeepSeek-V4-Pro across different question difficulty categories, highlighting win rates and tie percentages. Additionally, cost comparisons in terms of tool calls and token usage are provided. The document also compares DeepSeek-V4-Pro with its predecessor DeepSeek-V3.2 on various search Q&A tasks, breaking down results by objective and subjective question types and subcategories, with detailed win, loss, and tie statistics.", | |
| "text": "Appendix\nA. Author List and Acknowledgment\nA.1. Author List\nAuthors are listed alphabetically by their first name. Names marked with * denote individuals\nwho have departed from our team.\nResearch & Engineering:Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang*, Bingzheng Xu,\nBochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi\nDeng, Chenhao Xu, Chenze Shao, Chong Ruan*, Conner Sun, Damai Dai, Daya Guo*, Dejian\nYang, Deli Chen, Donghao Li, Erhang Li, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong\nDai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang,\nHanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei*, Haotian Yuan,\nHaowei Zhang*, Haowen Luo, Haoyu Chen, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi\nCao, Huazuo Gao, Hui Qu, Hui Zeng, J. Yang, J.Q. Zhu, Jia Yu, Jialiang Huang, Jiasheng Ye,\nJiashi Li, Jiaxin Xu, Jiewen Hu, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang\nYuan, Jingyuan Cheng, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran*, Junguang Jiang, Junjie Qiu,\nJunlong Li*, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang*,\nKuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao\nLuo, Linwang Ma, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, M.S. Di, M.Y Xu,\nMax Mei, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang,\nPeixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du,\nQiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan,\nRunji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S.H. Liu,\nShanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoyuan Chen,\nShengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng\nZhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye,\nTianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W. Zhang, Wangding\nZeng, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun\nGao, Wenkai Yang, Wenlve Huang, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen\nWang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie,\nXin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang,\nXu Chen, Xuanyu Wang, Xuecheng Su, Xuheng Lin, Xuwei Fu, Y.C. Yan, Y.Q. Wang*, Y.W. Ma,\nYanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Zhao,\nYaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang\nXiong, Ying He, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang,\nYixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo,\nYongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuhan\nWu, Yuhao Meng, Yuheng Zou, YuKun Li, Yunfan Xiong, Yupeng Chen, Yuqian Cao, Yuqian\nWang, Yushun Zhang, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan\nZhou, Yuyang Zhou, Yuzhen Huang, Z.F. Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zhangli\nSha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma,\nZhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhuoshu\nLi, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang*, Ziwei\nXie, Ziyi Gao, Zizheng Pan, Zongqing Yao.\nBusiness & Compliance:Chenchen Ling, Chengyu Hou, Dongjie Ji, Fang Wei, Hengqing Zhang,\nJia Luo, Jia Song, Jialu Cai, Jian Liang, Jiangting Zhou, Jieyu Yang, Jin Chen, Jingzi Zhou, Junmin\nZheng, Leyi Xia, Linyan Zhu, Miaojun Wang, Mingming Li, Minmin Han, Ning Wang, Panpan\n54Wang, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, W.L. Xiao, Wei An, Wenqing\nHou, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Xinyu Zhang, Xueyin Chen, Yao Xu, Yi\nShao, Yiling Ma, Ying Tang, Yuehan Yang, Yuer Xu, Yukun Zha, Yuping Lin, Yuting Yan, Zekai\nZhang, Zhe Ju, Zheren Gao, Zhongyu Wu, Zihua Qu, Ziyi Wan.\nA.2. Acknowledgment\nWe would like to thank Dolly Deng and other testers for their valuable suggestions and feedback\nregarding the capabilities of DeepSeek-V4 series models.\nB. Evaluation Details\nTable 9|Agentic Search vs. Retrieval Augmented Search for DeepSeek-V4-Pro.\nDifficulty Category # Agent Win RAG Win Tie Agent% RAG% Tie%\nEasyObjective Q&A (\u5ba2\u89c2\u95ee\u7b54) 196 110 43 43 56.1 21.9 21.9\nSubjective Q&A (\u4e3b\u89c2\u95ee\u7b54) 321 198 56 67 61.7 17.4 20.9\nHardObjective Q&A (\u5ba2\u89c2\u95ee\u7b54) 168 102 33 33 60.7 19.6 19.6\nSubjective Q&A (\u4e3b\u89c2\u95ee\u7b54) 184 126 27 31 68.5 14.7 16.8\nTotal (\u603b\u8ba1) 869 536 159 174 61.7 18.3 20.0\nTable 10|Cost Comparison:Agentic Search vs. Retrieval Augmented Search (Mean) for\nDeepSeek-V4-Pro. Most of the tool calls are parallel for Agentic Search.\nVersion Tool Calls Prefill (tokens) Output (tokens)\nV4 Agentic Search 16.2 13649 1526\nV4 Retrieval Augmented Search \u2014 10453 1308\nTable 11|Comparative Evaluation of DeepSeek-V4-Pro and DeepSeek-V3.2 on Search Q&A\nTasks.\nInternal Evaluation (\u5185\u90e8\u7efc\u5408\u8bc4\u4f30)\nCategory Subcategory # V4 win V3.2 win tie V4% V3.2% tie%\nObjective\nQ&A\n(\u5ba2\u89c2\u95ee\u7b54)Single-value Search (\u5355\u503c\u4fe1\u606f\u67e5\u627e) 95 36 10 49 37.9 10.5 51.6\nEntity Search (\u5b9e\u4f53\u4fe1\u606f\u67e5\u627e) 99 24 7 68 24.2 7.1 68.7\nEnumerative Search (\u679a\u4e3e\u578b\u4fe1\u606f\u67e5\u627e) 95 19 8 68 20.0 8.4 71.6\nSubtotal (\u5c0f\u8ba1) 289 79 25 185 27.3 8.7 64.0\nSubjective\nQ&A\n(\u4e3b\u89c2\u95ee\u7b54)Causal Analysis (\u539f\u56e0\u5206\u6790) 100 28 5 67 28.0 5.0 67.0\nComparison (\u5bf9\u6bd4) 96 28 20 48 29.2 20.8 50.0\nAdvice Seeking (\u5bfb\u6c42\u5efa\u8bae) 92 23 8 61 25.0 8.7 66.3\nRecommendation (\u63a8\u8350) 95 26 19 50 27.4 20.0 52.6\nPlanning & Strategy (\u653b\u7565\u8ba1\u5212) 92 32 11 49 34.8 12.0 53.3\nOpinion & Evaluation (\u8bc4\u4ef7\u770b\u6cd5) 96 30 8 58 31.2 8.3 60.4\nTrend Analysis (\u8d8b\u52bf\u5206\u6790) 96 23 3 70 24.0 3.1 72.9\nSubtotal (\u5c0f\u8ba1) 667 190 74 403 28.5 11.1 60.4\nTOTAL (\u603b\u8ba1) 956 269 99 588 28.1 10.4 61.5\n55" | |
| }, | |
| { | |
| "title": "Acknowledgment", | |
| "node_id": "0053", | |
| "start_index": 55, | |
| "end_index": 55, | |
| "summary": "The partial document primarily covers the evaluation and comparison of DeepSeek-V4 series models, particularly DeepSeek-V4-Pro, against previous versions and alternative search methods. It includes acknowledgments to contributors and testers, detailed performance metrics comparing Agentic Search and Retrieval Augmented Search across different question difficulty categories (objective and subjective Q&A), and cost analysis in terms of tool calls and token usage. Additionally, it presents a comparative internal evaluation of DeepSeek-V4-Pro versus DeepSeek-V3.2 on various search Q&A tasks, highlighting win rates, ties, and performance percentages across multiple subcategories of objective and subjective queries.", | |
| "text": "Wang, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, W.L. Xiao, Wei An, Wenqing\nHou, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Xinyu Zhang, Xueyin Chen, Yao Xu, Yi\nShao, Yiling Ma, Ying Tang, Yuehan Yang, Yuer Xu, Yukun Zha, Yuping Lin, Yuting Yan, Zekai\nZhang, Zhe Ju, Zheren Gao, Zhongyu Wu, Zihua Qu, Ziyi Wan.\nA.2. Acknowledgment\nWe would like to thank Dolly Deng and other testers for their valuable suggestions and feedback\nregarding the capabilities of DeepSeek-V4 series models.\nB. Evaluation Details\nTable 9|Agentic Search vs. Retrieval Augmented Search for DeepSeek-V4-Pro.\nDifficulty Category # Agent Win RAG Win Tie Agent% RAG% Tie%\nEasyObjective Q&A (\u5ba2\u89c2\u95ee\u7b54) 196 110 43 43 56.1 21.9 21.9\nSubjective Q&A (\u4e3b\u89c2\u95ee\u7b54) 321 198 56 67 61.7 17.4 20.9\nHardObjective Q&A (\u5ba2\u89c2\u95ee\u7b54) 168 102 33 33 60.7 19.6 19.6\nSubjective Q&A (\u4e3b\u89c2\u95ee\u7b54) 184 126 27 31 68.5 14.7 16.8\nTotal (\u603b\u8ba1) 869 536 159 174 61.7 18.3 20.0\nTable 10|Cost Comparison:Agentic Search vs. Retrieval Augmented Search (Mean) for\nDeepSeek-V4-Pro. Most of the tool calls are parallel for Agentic Search.\nVersion Tool Calls Prefill (tokens) Output (tokens)\nV4 Agentic Search 16.2 13649 1526\nV4 Retrieval Augmented Search \u2014 10453 1308\nTable 11|Comparative Evaluation of DeepSeek-V4-Pro and DeepSeek-V3.2 on Search Q&A\nTasks.\nInternal Evaluation (\u5185\u90e8\u7efc\u5408\u8bc4\u4f30)\nCategory Subcategory # V4 win V3.2 win tie V4% V3.2% tie%\nObjective\nQ&A\n(\u5ba2\u89c2\u95ee\u7b54)Single-value Search (\u5355\u503c\u4fe1\u606f\u67e5\u627e) 95 36 10 49 37.9 10.5 51.6\nEntity Search (\u5b9e\u4f53\u4fe1\u606f\u67e5\u627e) 99 24 7 68 24.2 7.1 68.7\nEnumerative Search (\u679a\u4e3e\u578b\u4fe1\u606f\u67e5\u627e) 95 19 8 68 20.0 8.4 71.6\nSubtotal (\u5c0f\u8ba1) 289 79 25 185 27.3 8.7 64.0\nSubjective\nQ&A\n(\u4e3b\u89c2\u95ee\u7b54)Causal Analysis (\u539f\u56e0\u5206\u6790) 100 28 5 67 28.0 5.0 67.0\nComparison (\u5bf9\u6bd4) 96 28 20 48 29.2 20.8 50.0\nAdvice Seeking (\u5bfb\u6c42\u5efa\u8bae) 92 23 8 61 25.0 8.7 66.3\nRecommendation (\u63a8\u8350) 95 26 19 50 27.4 20.0 52.6\nPlanning & Strategy (\u653b\u7565\u8ba1\u5212) 92 32 11 49 34.8 12.0 53.3\nOpinion & Evaluation (\u8bc4\u4ef7\u770b\u6cd5) 96 30 8 58 31.2 8.3 60.4\nTrend Analysis (\u8d8b\u52bf\u5206\u6790) 96 23 3 70 24.0 3.1 72.9\nSubtotal (\u5c0f\u8ba1) 667 190 74 403 28.5 11.1 60.4\nTOTAL (\u603b\u8ba1) 956 269 99 588 28.1 10.4 61.5\n55" | |
| } | |
| ] | |
| }, | |
| { | |
| "title": "Evaluation Details", | |
| "node_id": "0054", | |
| "start_index": 55, | |
| "end_index": 58, | |
| "summary": "The partial document primarily covers the evaluation and comparative analysis of various versions of the DeepSeek language models, particularly DeepSeek-V4-Pro, against previous versions (such as V3.2) and other models like Gemini-3.1-Pro and Claude-Opus-4.5. It includes detailed performance metrics across multiple categories, including objective and subjective Q&A tasks, Chinese functional writing, creative writing, and complex instruction following with multi-turn writing. The document presents quantitative results on win rates, ties, and percentages for different subcategories such as business, media, everyday, oral, official, academic, and creative writing styles. It also compares agentic search versus retrieval augmented search methods in terms of effectiveness and cost. Additionally, example outputs for specific tasks and acknowledgments for contributors are included. Overall, the document focuses on benchmarking DeepSeek models\u2019 capabilities, writing quality, instruction adherence, and cost efficiency across diverse linguistic and functional tasks.", | |
| "text": "Wang, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, W.L. Xiao, Wei An, Wenqing\nHou, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Xinyu Zhang, Xueyin Chen, Yao Xu, Yi\nShao, Yiling Ma, Ying Tang, Yuehan Yang, Yuer Xu, Yukun Zha, Yuping Lin, Yuting Yan, Zekai\nZhang, Zhe Ju, Zheren Gao, Zhongyu Wu, Zihua Qu, Ziyi Wan.\nA.2. Acknowledgment\nWe would like to thank Dolly Deng and other testers for their valuable suggestions and feedback\nregarding the capabilities of DeepSeek-V4 series models.\nB. Evaluation Details\nTable 9|Agentic Search vs. Retrieval Augmented Search for DeepSeek-V4-Pro.\nDifficulty Category # Agent Win RAG Win Tie Agent% RAG% Tie%\nEasyObjective Q&A (\u5ba2\u89c2\u95ee\u7b54) 196 110 43 43 56.1 21.9 21.9\nSubjective Q&A (\u4e3b\u89c2\u95ee\u7b54) 321 198 56 67 61.7 17.4 20.9\nHardObjective Q&A (\u5ba2\u89c2\u95ee\u7b54) 168 102 33 33 60.7 19.6 19.6\nSubjective Q&A (\u4e3b\u89c2\u95ee\u7b54) 184 126 27 31 68.5 14.7 16.8\nTotal (\u603b\u8ba1) 869 536 159 174 61.7 18.3 20.0\nTable 10|Cost Comparison:Agentic Search vs. Retrieval Augmented Search (Mean) for\nDeepSeek-V4-Pro. Most of the tool calls are parallel for Agentic Search.\nVersion Tool Calls Prefill (tokens) Output (tokens)\nV4 Agentic Search 16.2 13649 1526\nV4 Retrieval Augmented Search \u2014 10453 1308\nTable 11|Comparative Evaluation of DeepSeek-V4-Pro and DeepSeek-V3.2 on Search Q&A\nTasks.\nInternal Evaluation (\u5185\u90e8\u7efc\u5408\u8bc4\u4f30)\nCategory Subcategory # V4 win V3.2 win tie V4% V3.2% tie%\nObjective\nQ&A\n(\u5ba2\u89c2\u95ee\u7b54)Single-value Search (\u5355\u503c\u4fe1\u606f\u67e5\u627e) 95 36 10 49 37.9 10.5 51.6\nEntity Search (\u5b9e\u4f53\u4fe1\u606f\u67e5\u627e) 99 24 7 68 24.2 7.1 68.7\nEnumerative Search (\u679a\u4e3e\u578b\u4fe1\u606f\u67e5\u627e) 95 19 8 68 20.0 8.4 71.6\nSubtotal (\u5c0f\u8ba1) 289 79 25 185 27.3 8.7 64.0\nSubjective\nQ&A\n(\u4e3b\u89c2\u95ee\u7b54)Causal Analysis (\u539f\u56e0\u5206\u6790) 100 28 5 67 28.0 5.0 67.0\nComparison (\u5bf9\u6bd4) 96 28 20 48 29.2 20.8 50.0\nAdvice Seeking (\u5bfb\u6c42\u5efa\u8bae) 92 23 8 61 25.0 8.7 66.3\nRecommendation (\u63a8\u8350) 95 26 19 50 27.4 20.0 52.6\nPlanning & Strategy (\u653b\u7565\u8ba1\u5212) 92 32 11 49 34.8 12.0 53.3\nOpinion & Evaluation (\u8bc4\u4ef7\u770b\u6cd5) 96 30 8 58 31.2 8.3 60.4\nTrend Analysis (\u8d8b\u52bf\u5206\u6790) 96 23 3 70 24.0 3.1 72.9\nSubtotal (\u5c0f\u8ba1) 667 190 74 403 28.5 11.1 60.4\nTOTAL (\u603b\u8ba1) 956 269 99 588 28.1 10.4 61.5\n55Figure 14|Example output of a task that requires comparing two regular investment strategies\nfor the NASDAQ.\nFigure 15|Example output of a task which requires researching 2020-2025 Nobel Science Prizes\nand generating an analytical PDF report.\n56Table 12|Comparative Analysis of DeepSeek-V4-Pro and Gemini-3.1-Pro in Chinese Functional\nWriting.\nInternal Evaluation (\u5185\u90e8\u7efc\u5408\u8bc4\u4f30)\nCategory Subcategory # DS win Gem win Tie DS% Gem% Tie%\nBusiness\nWriting\n(\u529e\u516c\u6587\u672c)Report (\u62a5\u544a) 527 350 162 15 66.41 30.74 2.85\nProposal (\u65b9\u6848\u7b56\u5212) 291 181 103 7 62.20 35.40 2.41\nEducation (\u6559\u80b2\u57f9\u8bad) 159 100 56 3 62.89 35.22 1.89\nEmail & Letter (\u90ae\u4ef6\u4e66\u4fe1) 146 107 37 2 73.29 25.34 1.37\nNotice (\u901a\u77e5\u516c\u544a) 72 43 24 5 59.72 33.33 6.94\nProfessional (\u4e13\u4e1a\u6587\u672c) 63 34 27 2 53.97 42.86 3.17\nRecruitment (\u62db\u8058\u6c42\u804c) 42 27 15 0 64.29 35.71 0.00\nTechnical (\u6280\u672f\u6587\u672c) 29 22 7 0 75.86 24.14 0.00\nReview (\u4ecb\u7ecd\u8bc4\u4ef7) 20 15 5 0 75.00 25.00 0.00\nSubtotal (\u5c0f\u8ba1) 1349 879 436 34 65.16 32.32 2.52\nMedia\nWriting\n(\u5a92\u4f53\u6587\u672c)Social Media (\u793e\u4ea4\u5a92\u4f53\u6587\u6848) 267 156 101 10 58.43 37.83 3.75\nAd Copy (\u5e7f\u544a\u5546\u54c1\u6587\u6848) 214 109 98 7 50.93 45.79 3.27\nLong-form Content (\u5185\u5bb9\u5e73\u53f0\u957f\u6587) 99 71 25 3 71.72 25.25 3.03\nNews Report (\u65b0\u95fb\u62a5\u9053) 51 27 22 2 52.94 43.14 3.92\nAdvertorial (\u8425\u9500\u8f6f\u6587) 17 12 4 1 70.59 23.53 5.88\nHeadline (\u6807\u9898) 11 7 4 0 63.64 36.36 0.00\nNarration Script (\u53e3\u64ad\u6587\u6848) 4 2 1 1 50.00 25.00 25.00\nComment (\u8bc4\u8bba) 3 2 1 0 66.67 33.33 0.00\nSubtotal (\u5c0f\u8ba1) 666 386 256 24 57.96 38.44 3.60\nEveryday\nWriting\n(\u751f\u6d3b\u6587\u672c)Congratulatory (\u795d\u8d3a\u6587\u672c) 101 54 41 6 53.47 40.59 5.94\nCommunication (\u6c9f\u901a\u56de\u590d) 100 71 26 3 71.00 26.00 3.00\nReflection (\u5fc3\u5f97\u611f\u60f3) 90 68 17 5 75.56 18.89 5.56\nReview (\u4ecb\u7ecd\u8bc4\u4ef7) 55 44 9 2 80.00 16.36 3.64\nComment (\u8bc4\u8bba) 44 34 8 2 77.27 18.18 4.55\nSubtotal (\u5c0f\u8ba1) 390 271 101 18 69.49 25.90 4.62\nOral\nWriting\n(\u53e3\u5934\u6587\u672c)Speech (\u53d1\u8a00\u7a3f) 226 135 85 6 59.73 37.61 2.65\nNarration Script (\u53e3\u64ad\u6587\u6848) 51 25 23 3 49.02 45.10 5.88\nSales Script (\u8bdd\u672f) 31 22 6 3 70.97 19.35 9.68\nDialogue (\u5bf9\u8bdd\u6587\u672c) 10 4 6 0 40.00 60.00 0.00\nCongratulatory (\u795d\u8d3a\u6587\u672c) 1 1 0 0 100.00 0.00 0.00\nSubtotal (\u5c0f\u8ba1) 319 187 120 12 58.62 37.62 3.76\nOfficial\nDocument\n(\u516c\u6587\u6587\u672c)Administrative Doc (\u4e8b\u52a1\u6587\u4e66) 117 60 53 4 51.28 45.30 3.42\nPersonal Doc (\u4e2a\u4eba\u6587\u4e66) 73 45 27 1 61.64 36.99 1.37\nGovernment Doc (\u884c\u653f\u516c\u6587) 34 19 14 1 55.88 41.18 2.94\nSpeech (\u53d1\u8a00\u7a3f) 3 1 2 0 33.33 66.67 0.00\nEssay Writing (\u7533\u8bba\u5199\u4f5c) 3 1 1 1 33.33 33.33 33.33\nSubtotal (\u5c0f\u8ba1) 230 126 97 7 54.78 42.17 3.04\nAcademic\nWriting\n(\u5b66\u672f\u6587\u672c)Research Paper (\u5b66\u672f\u8bba\u6587) 104 67 32 5 64.42 30.77 4.81\nCoursework (\u8bfe\u7a0b\u4f5c\u4e1a) 90 53 35 2 58.89 38.89 2.22\nAcademic Support (\u5b66\u672f\u8f85\u52a9) 15 11 3 1 73.33 20.00 6.67\nScience Outreach (\u4e13\u4e1a\u79d1\u666e) 7 6 1 0 85.71 14.29 0.00\nSubtotal (\u5c0f\u8ba1) 216 137 71 8 63.43 32.87 3.70\nTotal (\u603b\u8ba1) 3170 1986 1081 103 62.65 34.10 3.25\n57Table 13|Comparative Analysis of DeepSeek-V4-Pro and Gemini-3.1-Pro in Chinese Creative\nWriting.\nInstruction Following(\u6307\u4ee4\u9075\u5faa) Writing Quality (\u5199\u4f5c\u8d28\u91cf)\nSubcategory (\u6587\u4f53) # DS Gem Tie DS% Gem% Tie% DS Gem Tie DS% Gem% Tie%\nFiction (\u5c0f\u8bf4\u6545\u4e8b) 836 504 323 5 60.58 38.82 0.60 672 157 3 80.77 18.87 0.36\nGeneral Fiction (\u6cdb\u5c0f\u8bf4\u6545\u4e8b) 662 368 290 3 55.67 43.87 0.45 467 194 0 70.65 29.35 0.00\nFan Fiction (\u540c\u4eba\u6587) 410 253 150 3 62.32 36.95 0.74 338 67 1 83.25 16.50 0.25\nGeneral Fan Fic. (\u6cdb\u540c\u4eba\u6587) 202 111 90 1 54.95 44.55 0.50 161 40 1 79.70 19.80 0.50\nNarrative (\u8bb0\u53d9\u6587) 171 115 54 2 67.25 31.58 1.17 141 30 0 82.46 17.54 0.00\nGeneral Prose (\u6cdb\u6563\u6587) 124 83 40 1 66.94 32.26 0.81 88 36 0 70.97 29.03 0.00\nProse (\u6563\u6587) 112 74 38 0 66.07 33.93 0.00 92 20 0 82.14 17.86 0.00\nWriting Style (\u6587\u7b14) 112 81 31 0 72.32 27.68 0.00 86 26 0 76.79 23.21 0.00\nClassical Poetry (\u53e4\u8bd7\u6587) 48 24 24 0 50.00 50.00 0.00 39 9 0 81.25 18.75 0.00\nModern Poetry (\u73b0\u4ee3\u8bd7) 43 23 20 0 53.49 46.51 0.00 32 11 0 74.42 25.58 0.00\nLyrics (\u6b4c\u8bcd) 30 8 22 0 26.67 73.33 0.00 16 14 0 53.33 46.67 0.00\nLiterary Appreciation (\u8d4f\u6790) 27 20 7 0 74.07 25.93 0.00 18 9 0 66.67 33.33 0.00\nGeneral Argument. (\u6cdb\u8bae\u8bba\u6587) 24 15 9 0 62.50 37.50 0.00 17 7 0 70.83 29.17 0.00\nGeneral Narrative (\u6cdb\u8bb0\u53d9\u6587) 23 11 12 0 47.83 52.17 0.00 15 8 0 65.22 34.78 0.00\nGeneral Classical (\u6cdb\u53e4\u6587\u8bd7\u6b4c) 9 5 4 0 55.56 44.44 0.00 5 4 0 55.56 44.44 0.00\nCreative Writing (\u521b\u610f\u5199\u4f5c) 6 2 4 0 33.33 66.67 0.00 4 2 0 66.67 33.33 0.00\nArgumentative (\u8bae\u8bba\u6587) 5 5 0 0 100.00 0.00 0.00 5 0 0 100.00 0.00 0.00\nGeneral Mod. Poetry (\u6cdb\u73b0\u4ee3\u8bd7) 2 1 1 0 50.00 50.00 0.00 2 0 0 100.00 0.00 0.00\nTotal (\u603b\u8ba1) 2837 1703 1119 15 60.03 39.44 0.53 2198 634 5 77.48 22.35 0.18\nTable 14|DeepSeek-V4-Pro vs. Claude-Opus-4.5 on Complex Instruction Following and Multi-\nTurn Writing.\nInternal Evaluation (\u5185\u90e8\u7efc\u5408\u8bc4\u4f30)\nCategory # DS Opus Tie DS% Opus% Tie%\nComplex Inst. Following (\u590d\u6742\u6307\u4ee4\u8ddf\u968f) 49 23 26 0 46.9% 53.1% 0.0%\nMulti-Turn Writing (\u591a\u8f6e\u5199\u4f5c) 147 67 76 4 45.6% 51.7% 2.7%\nTotal (\u603b\u8ba1) 196 90 102 4 45.9% 52.0% 2.0%\n58" | |
| } | |
| ] | |
| } |