Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies

• URL: http://arxiv.org/abs/2501.18817v1
• Date: Fri, 31 Jan 2025 00:28:29 GMT
• Title: Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
• Abstract summary: We propose two approaches to enhance the reasoning ability of less resource-intensive models.<n>One is to provide them with a generalised strategy for solving tasks within a given domain, generated by a more resource-intensive model.<n>The other is to exploit their cost-effectiveness by iteratively prompting these models to correct errors in their proposed solutions.
• Score: 0.9093413254392775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in the reasoning skills of Large Language Models (LLMs) demonstrate an increase in the ability of LLMs to solve simple planning tasks. However, as long as the driving force behind improved reasoning capability is the size and complexity of the model, the financial and computational costs associated with running them will also increase. This trend raises questions about continued accessibility and whether these improvements will increase at the same pace as models continue to grow in size and expense. We propose two approaches to enhance the reasoning ability of less resource-intensive LLMs. (1) Provide them with a generalised strategy for solving tasks within a given domain, generated by a more resource-intensive LLM. (2) Exploit their cost-effectiveness by iteratively prompting these models to correct errors in their proposed solutions. Our empirical results from planning and mathematical reasoning tasks demonstrate that these methods improve the performance of less resource-intensive LLMs to levels comparable with their more resource-intensive counterparts, at a fraction of the cost. Additionally, we show that the utilisation of generalised strategies in our experiments reduced the cost of the less resource-intensive model by nearly 30 percent on average.

Related papers

• rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection [49.74493901036598]
  Large language models (LLMs) are post-trained through reinforcement learning (RL) to evolve into Reasoning Language Models (RLMs)<n>This paper proposes a novel reinforced strategy injection mechanism (rSIM) that enables any LLM to become an RLM by employing a small planner.<n> Experimental results show that rSIM enables Qwen2.5-0.5B to become an RLM and significantly outperform Qwen2.5-14B.
  arXiv  Detail & Related papers  (2025-12-09T06:55:39Z)
• More with Less: An Empirical Study of Turn-Control Strategies for Efficient Coding Agents [4.980051859336524]
  Coding agents operate in iterative loops (turns) to solve software engineering tasks.<n>They are becoming increasingly powerful, but their practical deployment is hindered by significant and unpredictable costs.<n>We show that a fixed-turn limit, specifically at the 75th percentile of the baseline, serves as a "sweet spot"<n>We then show that a fixed-turn strategy consistently outperforms fixed-limit approaches, achieving comparable or better solve rates while further reducing costs by an additional 12%-24% by intelligently allocating resources only to tasks that need them.
  arXiv  Detail & Related papers  (2025-10-19T10:32:18Z)
• BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens [33.607723102172194]
  BudgetThinker is a framework designed to empower Large Language Models with budget-aware reasoning.<n>We show that BudgetThinker significantly surpasses strong baselines in maintaining performance across a variety of reasoning budgets.
  arXiv  Detail & Related papers  (2025-08-24T03:17:50Z)
• PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving [66.42260489147617]
  We introduce PLAN-TUNING, a framework that distills synthetic task decompositions from large-scale language models.<n>Plan-TUNING fine-tunes smaller models via supervised and reinforcement-learning objectives to improve complex reasoning.<n>Our analysis demonstrates how planning trajectories improves complex reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-10T07:30:44Z)
• Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs [45.83245433138508]
  Large language models (LLMs) have rapidly progressed into general-purpose agents capable of solving a broad spectrum of tasks.<n>They apply fixed inference-time compute regardless of task complexity, often overthinking simple problems while underthinking hard ones.<n>This survey presents a comprehensive review of efficient test-time compute strategies, which aim to improve the computational efficiency of LLM reasoning.
  arXiv  Detail & Related papers  (2025-07-02T18:27:42Z)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL [62.984693936073974]
  Large language models (LLMs) excel in tasks like question answering and dialogue.<n>Complex tasks requiring interaction, such as negotiation and persuasion, require additional long-horizon reasoning and planning.<n>We propose a novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents.
  arXiv  Detail & Related papers  (2025-05-23T16:51:54Z)
• Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation [19.48826538310603]
  We introduce LVLM to Policy (LVLM2P), a framework that distills knowledge from large vision-language models (LVLM) into more efficientReinforcement Learning agents.<n>Our approach leverages the LVLM as a teacher, providing instructional actions based on trajectories collected by the RL agent.<n>We show that LVLM2P significantly enhances the sample efficiency of baseline RL algorithms.
  arXiv  Detail & Related papers  (2025-05-16T13:15:54Z)
• Lightweight Latent Verifiers for Efficient Meta-Generation Strategies [0.5892638927736115]
  Verifiers are auxiliary models that assess the correctness of outputs generated by base large language models (LLMs) In this work, we introduce a novel lightweight verification approach, LiLaVe, which reliably extracts correctness signals from the hidden states of the base LLM. A key advantage of LiLaVe is its ability to operate with only a small fraction of the computational budget required by traditional LLM-based verifiers.
  arXiv  Detail & Related papers  (2025-04-23T14:33:20Z)
• A Survey of Scaling in Large Language Model Reasoning [62.92861523305361]
  We provide a comprehensive examination of scaling in large Language models (LLMs) reasoning. We analyze scaling in reasoning steps that improves multi-step inference and logical consistency. We discuss scaling in training-enabled reasoning, focusing on optimization through iterative model improvement.
  arXiv  Detail & Related papers  (2025-04-02T23:51:27Z)
• Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation [29.579349371114702]
  Direct Preference Optimization (DPO) is a cost-effective alternative to reinforcement learning (RL) for large language models (LLMs) We show that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance. With simple verifiable rewards, our model achieves RL-level performance with significantly lower computational overhead.
  arXiv  Detail & Related papers  (2025-03-17T06:28:25Z)
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning [87.30285670315334]
  textbfR1-Searcher is a novel two-stage outcome-based RL approach designed to enhance the search capabilities of Large Language Models. Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start. Our experiments demonstrate that our method significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini.
  arXiv  Detail & Related papers  (2025-03-07T17:14:44Z)
• Universal Model Routing for Efficient LLM Inference [72.65083061619752]
  We consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. We prove that these strategies are estimates of a theoretically optimal routing rule, and provide an excess risk bound to quantify their errors.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems [1.430963201405577]
  Large Language Model (LLM)-based systems are usually designed with a single, general-purpose LLM to handle all user queries.<n>These systems may be inefficient as different queries may require different levels of reasoning, domain knowledge or pre-processing.<n>A routing mechanism can therefore be employed to route queries to more appropriate components, such as smaller or specialised models.
  arXiv  Detail & Related papers  (2025-02-01T12:08:38Z)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs [22.177654792824896]
  We focus on small-sized language models (3B to 7B parameters) for their cost-efficiency and accessibility.<n>We explore various training configurations and strategies across four open-source pre-trained models.<n>Key insights from our work include: (i) larger batch sizes paired with lower learning rates lead to improved model performance on benchmarks such as MMLU, MTBench, and Open LLM Leaderboard; (ii) early-stage training dynamics, such as lower gradient norms and higher loss values, are strong indicators of better final model performance; (iv) we observed no significant difference in performance between phased and stacked training strategies, but
  arXiv  Detail & Related papers  (2024-12-17T21:16:59Z)
• Reinforcement Learning Enhanced LLMs: A Survey [45.57586245741664]
  We will make a systematic review of the most up-to-date state of knowledge on RL-enhanced large language models (LLMs) Specifically, we detail the basics of RL; (2) introduce popular RL-enhanced LLMs; (3) review researches on two widely-used reward model-based RL techniques: Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF)
  arXiv  Detail & Related papers  (2024-12-05T16:10:42Z)
• Efficient Reinforcement Learning with Large Language Model Priors [18.72288751305885]
  Large language models (LLMs) have recently emerged as powerful general-purpose tools. We propose treating LLMs as prior action distributions and integrating them into RL frameworks. We show that incorporating LLM-based action priors significantly reduces exploration and complexity optimization.
  arXiv  Detail & Related papers  (2024-10-10T13:54:11Z)
• CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks [2.9449838351181374]
  Post-training, particularly reinforcement learning, has become a new learning paradigm for large language models (LLMs) We propose searching within the action space on high-level abstract plans to enhance model generalization. We show that our method, trained exclusively on GSM8K and MATH, significantly improves performance.
  arXiv  Detail & Related papers  (2024-09-13T08:59:31Z)
• Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning [53.6472920229013]
  Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks. LLMs are prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning. We introduce Q*, a framework for guiding LLMs decoding process with deliberative planning.
  arXiv  Detail & Related papers  (2024-06-20T13:08:09Z)
• Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning [28.077228879886402]
  Reinforcement Learning (RL) suffers from sample inefficiency in reward domains, and the problem is further pronounced in case of transitions. To improve the sample efficiency, reward shaping is a well-studied approach to introduce intrinsic rewards that can help the RL agent converge to an optimal policy faster.
  arXiv  Detail & Related papers  (2024-05-24T03:53:57Z)
• Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection [80.63946798650653]
  Decision centers on whether to use a large LLM with better performance or a smaller one with reduced costs. We propose a simpler solution; we use only the uncertainty of the generations of the small LLM as the decision criterion. Our experiments reveal this simple solution optimally balances cost and performance, outperforming existing methods on 25 out of 27 experimental setups.
  arXiv  Detail & Related papers  (2024-05-03T14:38:59Z)
• Thought of Search: Planning with Language Models Through The Lens of Efficiency [22.47015814897628]
  We argue that recent trends abandon both soundness and completeness for the sake of inefficiency. We show that by using LLMs to produce the code for the search components we can solve the entire datasets with 100% accuracy.
  arXiv  Detail & Related papers  (2024-04-18T01:27:29Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs [60.40396361115776]
  This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in large language models (LLMs) with a slim proxy model. We employ a proxy model which has far fewer parameters, and take its answers as answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM.
  arXiv  Detail & Related papers  (2024-02-19T11:11:08Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.