HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression

• URL: http://arxiv.org/abs/2606.01934v1
• Date: Mon, 01 Jun 2026 09:01:55 GMT
• Title: HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression
• Abstract summary: HMPO is a cost-effective, single-stage reinforcement learning framework.<n>It generalizes seamlessly across math, code, science, and instruction-following tasks.<n>It achieves 19%--46% token compression with negligible accuracy degradation.
• Score: 32.10638394568435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models achieve remarkable performance via extended chain-of-thought (CoT) reasoning, yet this lengthy process incurs substantial inference overhead. Existing CoT compression methods struggle with inflexible manual length budgets, computationally expensive multi-stage training pipelines, and fragile scalability restricted to small models. We propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework. HMPO efficiently compresses CoT via three synergistic components: an adaptive median-based budget derived from successful rollouts to eliminate manual tuning, a cosine-decay token reward for smooth length penalization, and a multiplicative reward formulation that substantially mitigates trivial reward hacking by strictly prioritizing answer correctness. Trained exclusively on mathematical data, HMPO generalizes seamlessly across math, code, science, and instruction-following tasks. Extensive experiments scaling from 9B to 122B parameters across dense and Mixture-of-Experts (MoE) architectures demonstrate that HMPO achieves 19%--46% token compression with negligible accuracy degradation, all while drastically reducing training costs compared to existing multi-stage baselines.

Related papers

• MCPO: Modality-Contrastive Preference Optimization for Multimodal Chain-of-Thought Compression [40.580984703096945]
  We propose a two-stage length-compression method that requires fewer than 900 training samples.<n>In the compression stage, we introduce a step-level Normalized Cross-Modal Mutual Information pruning algorithm.<n>In the alignment stage, we use an asymmetric multimodal length-controlled preference loss.<n>Experiments show that our method can reduce CoT length by up to 69.5% and achieve up to 3.34x end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-09-04T09:54:53Z)
• ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression [14.830831997228657]
  Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency.<n>We propose textbfExpThinkxspace, an RL framework that addresses both dimensions through two complementary mechanisms.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that textbfExpThinkxspace reduces average response length by up to 77% while simultaneously improving accuracy.
  arXiv  Detail & Related papers  (2026-05-08T09:37:30Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog [72.4168434368873]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources.<n>We propose a gradual compacting method that divides the compression process into multiple fine-grained iterations.<n>This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss.
  arXiv  Detail & Related papers  (2026-02-04T06:56:52Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinement [21.015658035494514]
  Chain-of-Thought (CoT) reasoning improves performance on complex tasks but introduces significant latency due to verbosity.<n>We propose Multiround Adaptive Chain-of-Thought Compression (MACC)<n>Our method achieves an average accuracy improvement of 5.6 percent over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-09-26T10:02:48Z)
• Training Long-Context LLMs Efficiently via Chunk-wise Optimization [60.05884946552877]
  We present textitSequential Chunk-wise Optimization (SeCO), a memory-efficient training paradigm that partitions lengthy inputs into manageable chunks.<n>We also introduce textitSparse Chunk-wise Optimization (SpaCO), which reduces computational overhead by selectively propagating gradients to specific chunks.<n>SpaCO decouples the computational cost of backpropagation from the context length, enabling training time to gradually converge to inference time as sequences become longer.
  arXiv  Detail & Related papers  (2025-05-22T14:11:34Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.