Doing More With Less: Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling

• URL: http://arxiv.org/abs/2604.25098v1
• Date: Tue, 28 Apr 2026 01:04:09 GMT
• Title: Doing More With Less: Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
• Abstract summary: Previous work has shown that structured pruning (methods which removes entire set of layer blocks) significantly degrades TTS reasoning performance.<n>In this work, we investigate whether unstructured pruning (methods that carefully remove only certain redundant/detrimental weights) exhibits similar limitations.<n>Surprisingly, our experiments consistently show that unstructured pruning augments TTS performance compared to structured pruning.
• Score: 9.094127664014627
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While current Large Language Models (LLMs) exhibit remarkable reasoning capabilities through test-time compute scaling (TTS), their massive parameter counts and high inference costs have motivated the development of pruning methods that can reduce model size without sacrificing performance. However, specific to reasoning LLMs, prior work has shown that structured pruning (methods which removes entire set of layer blocks), significantly degrades TTS reasoning performance. In this work, we revisit this assumption and instead investigate whether unstructured pruning (methods that carefully remove only certain redundant/detrimental weights) exhibits similar limitations. Surprisingly, our extensive experiments across four reasoning benchmarks on two reasoning LLMs: s1.1-7B and Qwen3-8B, consistently show that unstructured pruning augments TTS performance compared to structured pruning, and at times can even outperform the unpruned full-weight LLMs. Furthermore, we also empirically study the impact of different layer-wise sparsity allocation strategies, which are an important parametric choice for instantiating unstructured pruning methods. These findings challenge the conventional notion that pruning always reduces TTS performance and in fact, suggest that carefully undertaken pruning can improve TTS effectiveness even further.

Related papers

• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn [58.439517684779936]
  In-context learning (ICL) adapts large language models to new tasks by conditioning on demonstrations in the prompt without parameter updates.<n>We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer.
  arXiv  Detail & Related papers  (2026-05-13T13:30:12Z)
• Efficient and Stable Reinforcement Learning for Diffusion Language Models [59.75789436018925]
  Reinforcement Learning (RL) is crucial for unlocking the complex reasoning capabilities of Diffusion-based Large Language Models (dLLMs)<n>Applying to dLLMs faces unique challenges in efficiency and stability.<n>We propose Spatio-Temporal Pruning (STP), a framework designed to simultaneously improve the efficiency and stability of RL for dLLMs.
  arXiv  Detail & Related papers  (2026-02-09T17:04:23Z)
• Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog [72.4168434368873]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources.<n>We propose a gradual compacting method that divides the compression process into multiple fine-grained iterations.<n>This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss.
  arXiv  Detail & Related papers  (2026-02-04T06:56:52Z)
• A State-Transition Framework for Efficient LLM Reasoning [58.18141262230392]
  Long Chain-of-Thought (CoT) reasoning significantly improves Large Language Models (LLMs) performance on complex reasoning tasks.<n>Existing studies usually enhance the reasoning efficiency of LLMs by compressing CoT sequences.<n>We propose an efficient reasoning framework that models the reasoning process of LLMs as a state-transition process.
  arXiv  Detail & Related papers  (2026-02-01T12:40:40Z)
• When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs [40.79077285268906]
  We study the impact of layer pruning on long-chain reasoning through the lens of test-time scaling.<n>We demonstrate that pruning even one or two layers can severely impair test-time scaling.<n>These findings call for a rethinking of layer pruning strategies.
  arXiv  Detail & Related papers  (2025-10-25T09:22:22Z)
• Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistency [3.6199690908942546]
  Self-Consistency (SC) generates multiple reasoning chains in parallel and selects the final answer via majority voting.<n>We propose Slim-SC, a step-wise pruning strategy that identifies and removes redundant chains using inter-chain similarity at the thought level.<n> Experiments show that Slim-SC reduces latency and KVC usage by up to 45% and 26%, respectively, with R1-Distill.
  arXiv  Detail & Related papers  (2025-09-17T14:00:51Z)
• Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning [15.137717200618454]
  Prune-on-Logic is a framework that transforms Long-CoT into logic graphs and selectively prunes low-utility reasoning steps.<n>We find that verification pruning consistently improves accuracy while reducing token usage, whereas reasoning or indiscriminate pruning degrades performance.
  arXiv  Detail & Related papers  (2025-05-20T16:38:32Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning [108.07030347318624]
  We show that scaling with longer Chain of Thoughts (CoTs) can indeed impair the reasoning performance of Large Language Models (LLMs) in certain domains.<n>We propose a Thinking- Optimal Scaling strategy to teach models to adopt different reasoning efforts for deep thinking.<n>Our self-improvement models built upon Qwen2.5-32B-Instruct outperform other distillation-based 32B o1-like models across various math benchmarks.
  arXiv  Detail & Related papers  (2025-02-25T10:48:05Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• When More is Less: Understanding Chain-of-Thought Length in LLMs [51.631483479081645]
  Large Language Models (LLMs) employ Chain-of-Thought (CoT) reasoning to deconstruct complex problems.<n>This paper argues that longer CoTs are often presumed superior, arguing that longer is not always better.
  arXiv  Detail & Related papers  (2025-02-11T05:28:59Z)
• SlimGPT: Layer-wise Structured Pruning for Large Language Models [15.252798256418279]
  Batched Greedy Pruning for rapid and near-optimal pruning.<n>Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation.<n> Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-12-24T02:49:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.