Dream-Coder 7B: An Open Diffusion Language Model for Code

• URL: http://arxiv.org/abs/2509.01142v1
• Date: Mon, 01 Sep 2025 05:30:56 GMT
• Title: Dream-Coder 7B: An Open Diffusion Language Model for Code
• Authors: Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, Lingpeng Kong,
• Abstract summary: We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities.<n>Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task.
• Score: 99.14959222355988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.

Related papers

• Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model [35.59660517313579]
  Diffusion-based language models (DLLMs) offer non-sequential, blockwise generation and richer data reuse compared to autoregressive (AR) models.<n>We introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline.
  arXiv  Detail & Related papers  (2026-01-22T12:13:17Z)
• CoDA: Coding LM via Diffusion Adaptation [102.62730448092888]
  CoDA pairs large-scale diffusion pre-training with code-centric mid-training and instruction tuning.<n>On Humaneval, MBPP, and EvalPlus, CoDA-1.7B-Instruct matches or surpasses diffusion models up to 7B parameters.
  arXiv  Detail & Related papers  (2025-09-27T05:41:55Z)
• Dream 7B: Diffusion Large Language Models [85.26033751898296]
  We introduce Dream 7B, the most powerful open diffusion large language model to date.<n>Our model consistently outperforms existing diffusion language models on general, mathematical, and coding tasks.
  arXiv  Detail & Related papers  (2025-08-21T12:09:58Z)
• DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation [68.19756761027351]
  Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models.<n>We investigate their denoising processes and reinforcement learning methods.<n>Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework.
  arXiv  Detail & Related papers  (2025-06-25T17:35:47Z)
• MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings [2.1262605464247812]
  Self-Distillation is a principled approach to trading inference cost for accuracy across various code understanding tasks.<n>Our architecture improves text-to-code and code-to-code search by targeting specific encoder layers as exit heads.<n>We release a new dataset created through code translation that extends text-to-code benchmarks with cross-language code-to-code pairs.
  arXiv  Detail & Related papers  (2025-03-04T21:08:17Z)
• Toward Exploring the Code Understanding Capabilities of Pre-trained Code Generation Models [12.959392500354223]
  We pioneer the transfer of knowledge from pre-trained code generation models to code understanding tasks. We introduce CL4D, a contrastive learning method designed to enhance the representation capabilities of decoder-only models.
  arXiv  Detail & Related papers  (2024-06-18T06:52:14Z)
• Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach [66.51005288743153]
  We investigate the legal and ethical issues of current neural code completion models. We tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks. We evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models.
  arXiv  Detail & Related papers  (2024-04-22T15:54:53Z)
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning [92.36705236706678]
  "CodeRL" is a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning. During inference, we introduce a new generation procedure with a critical sampling strategy. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives.
  arXiv  Detail & Related papers  (2022-07-05T02:42:15Z)
• InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees [17.461451218469062]
  This paper proposes InferCode to overcome the limitation by adapting the self-language learning mechanism to build source code model. Subtrees in ASTs are treated with InferCode as the labels for training code representations without any human labeling effort or the overhead of expensive graph construction. Compared to previous code learning techniques applied to the same downstream tasks, such as Code2Vec, Code2Seq, ASTNN, higher performance results are achieved using our pre-trained InferCode model.
  arXiv  Detail & Related papers  (2020-12-13T10:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.