DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

• URL: http://arxiv.org/abs/2606.19348v1
• Date: Sun, 26 Apr 2026 14:49:33 GMT
• Title: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Abstract summary: We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models.<n>DeepSeek-V4 series incorporate several key upgrades in architecture and optimization.<n>DeepSeek-V4 series are highly efficient in long-context scenarios.
• Score: 240.40422202055993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) -- both supporting a context length of one million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold-Constrained Hyper-Connections (mHC) that enhance conventional residual connections; (3) and the Muon optimizer for faster convergence and greater training stability. We pre-train both models on more than 32T diverse and high-quality tokens, followed by a comprehensive post-training pipeline that unlocks and further enhances their capabilities. DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks. Meanwhile, DeepSeek-V4 series are highly efficient in long-context scenarios. In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2. This enables us to routinely support one-million-token contexts, thereby making long-horizon tasks and further test-time scaling more feasible. The model checkpoints are available at https://huggingface.co/collections/deepseek-ai/deepseek-v4.

Related papers

• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation [84.86814271505109]
  LongLive-2.0 is an NVFP4-based parallel infrastructure throughout the full training and inference workflow of long video generation.<n>For training, we introduce sequence-parallel autoregressive (AR) training, instantiated as Balanced SP.<n>Experiments show up to 2.15x speedup in training, and 1.84x in inference.
  arXiv  Detail & Related papers  (2026-05-18T17:57:03Z)
• Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding [144.70522923640095]
  Large language models (LLMs) face low hardware efficiency during decoding.<n>This paper introduces Step-3, a hardware-aware model-system co-design optimized for minimizing decoding costs.<n>Step-3 significantly reduces theoretical decoding costs compared with models like DeepSeek-V3 and Qwen3 MoE 235B.
  arXiv  Detail & Related papers  (2025-07-25T16:53:13Z)
• DeepSeek-V3 Technical Report [147.16121855209246]
  We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.<n>We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages.<n> Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models.
  arXiv  Detail & Related papers  (2024-12-27T04:03:16Z)
• DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding [39.14141055325595]
  We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models.<n>For the vision component, we incorporate a dynamic tiling vision encoding strategy designed for processing high-resolution images with different aspect ratios.<n>For the language component, we leverage DeepSeekMoE models with the Multi-head Latent Attention mechanism.
  arXiv  Detail & Related papers  (2024-12-13T17:37:48Z)
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence [43.589403386634615]
  DeepSeek-Coder-V2 is an open-source code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. DeepSeek-Coder-V2 is further pre-trained from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion tokens. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves superior performance compared to closed-source models such as GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro.
  arXiv  Detail & Related papers  (2024-06-17T13:51:35Z)
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model [118.06260386652778]
  We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs.
  arXiv  Detail & Related papers  (2024-05-07T15:56:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.