ZAYA1-8B Technical Report

• URL: http://arxiv.org/abs/2605.05365v1
• Date: Wed, 06 May 2026 18:44:08 GMT
• Title: ZAYA1-8B Technical Report
• Abstract summary: We present ZAYA1-8B, a mixture-of-experts (MoE) model with 700M active and 8B total parameters.<n>ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks.<n>Post-training uses a four-stage RL cascade: reasoning warmup on math and puzzles; a 400-task RLVE-Gym curriculum; math and code RL with test-time compute traces and synthetic code environments.
• Score: 14.894881882111605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining, midtraining, and supervised fine-tuning (SFT) were performed on a full-stack AMD compute, networking, and software platform. With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, and remains competitive with substantially larger open-weight reasoning models. ZAYA1-8B was trained from scratch for reasoning, with reasoning data included from pretraining onward using an answer-preserving trimming scheme. Post-training uses a four-stage RL cascade: reasoning warmup on math and puzzles; a 400-task RLVE-Gym curriculum; math and code RL with test-time compute traces and synthetic code environments built from competitive-programming references; and behavioral RL for chat and instruction following. We also introduce Markovian RSA, a test-time compute method that recursively aggregates parallel reasoning traces while carrying forward only bounded-length reasoning tails between rounds. In TTC evaluation, Markovian RSA raises ZAYA1-8B to 91.9\% on AIME'25 and 89.6\% on HMMT'25 while carrying forward only a 4K-token tail, narrowing the gap to much larger reasoning models including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High.

Related papers

• Mellum2 Technical Report [1.1239988514721222]
  Mellum 2 is a general-purpose language model specialized in software engineering.<n>It is the successor to the completion-focused 4B dense Mellum model.<n>Mellum 2 is competitive with open-weight baselines in the 4B-14B range while running at the per-token compute of a 2.5B dense model.
  arXiv  Detail & Related papers  (2026-05-29T13:01:11Z)
• Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming [10.478834409140623]
  We study how to scale reasoning token budgets for competitive programming through two complementary approaches.<n>We introduce a multi-round parallel thinking pipeline that distributes the token budget across threads and rounds of generation, verification, and refinement.<n>We train the model end-to-end on this pipeline to match the training objective to the test-time structure.
  arXiv  Detail & Related papers  (2026-04-01T18:05:49Z)
• PRISM: Demystifying Retention and Interaction in Mid-Training [20.198164159173647]
  PRISM is a comprehensive empirical study of mid-training design choices for large language models.<n>We show that mid-training on approximately 27B high-quality tokens yields consistent gains of +15 to +40 points on math, +5 to +12 points on code, and +6 to +13 points on science benchmarks while preserving general performance.
  arXiv  Detail & Related papers  (2026-03-17T19:04:33Z)
• PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning [62.55836470496192]
  PaCoRe is a training-and-inference framework designed to overcome a central limitation of contemporary language models.<n>PaCoRe departs from the traditional sequential paradigm by driving TTC through massive parallel exploration coordinated via a message-passing architecture in multiple rounds.<n>Trained end-to-end with large-scale, outcome-based reinforcement learning, the model masters the synthesis abilities required by PaCoRe and scales to multi-million-token effective TTC without exceeding context limits.
  arXiv  Detail & Related papers  (2026-01-09T07:24:43Z)
• Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation [43.29267000439331]
  Large language models (LLMs) have demonstrated impressive reasoning capabilities when provided with chain-of-thought exemplars.<n>In this work, we introduce Prompting Test-Time Scaling (P-TTS), a simple yet effective inference-time data augmentation strategy.
  arXiv  Detail & Related papers  (2025-10-10T17:57:04Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning [65.68667585027232]
  Parallel thinking is a novel approach for enhancing the reasoning capabilities of large language models.<n>We propose textbfParallel-R1, the first reinforcement learning framework that enables parallel thinking behaviors.<n>Our framework employs a progressive curriculum that explicitly addresses the cold-start problem in training parallel thinking.
  arXiv  Detail & Related papers  (2025-09-09T17:59:35Z)
• LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters! [53.84130385074551]
  Large reasoning models (LRMs) tackle complex reasoning problems by following long chain-of-thoughts (Long CoT)<n>We find that a Large Language model (LLM) can effectively learn Long CoT reasoning through data-efficient supervised fine-tuning (SFT) and parameter-efficient low-rank adaptation (LoRA)<n>With just 17k long CoT training samples, the Qwen2.5-32B-Instruct model achieves significant improvements on a wide range of math and coding benchmarks.
  arXiv  Detail & Related papers  (2025-02-11T08:48:48Z)
• Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning [65.2421542320293]
  Reasoning abilities are crucial components of general intelligence.<n>Recent advances by proprietary companies, such as o-series models of OpenAI, have made remarkable progress on reasoning tasks.<n>This paper proposes a new RL framework, termed OREAL, to pursue the performance limit that can be achieved through textbfOutcome textbfREwtextbfArd-based reinforcement textbfLearning for mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2025-02-10T18:57:29Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.