Dynamic Strategy Planning for Efficient Question Answering with Large Language Models

• URL: http://arxiv.org/abs/2410.23511v2
• Date: Sat, 08 Feb 2025 00:48:39 GMT
• Title: Dynamic Strategy Planning for Efficient Question Answering with Large Language Models
• Abstract summary: We propose a novel technique DyPlan, to induce a dynamic strategy selection process in Large Language Models (LLMs)<n>DyPlan incorporates an initial decision step to select the most suitable strategy conditioned on the input question and guides the LLM's response generation accordingly.<n>Experiments reveal how DyPlan can improve model performance by 7-13% while reducing the cost by 11-32% relative to the best baseline model.
• Score: 41.34792517255731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Research has shown the effectiveness of reasoning (e.g., Chain-of-Thought), planning (e.g., SelfAsk), and retrieval augmented generation strategies to improve the performance of Large Language Models (LLMs) on various tasks, such as question answering. However, using a single fixed strategy to answer different kinds of questions is suboptimal in performance and inefficient in terms of generated output tokens and performed retrievals. In our work, we propose a novel technique DyPlan, to induce a dynamic strategy selection process in LLMs, to improve performance and reduce costs in question-answering. DyPlan incorporates an initial decision step to select the most suitable strategy conditioned on the input question and guides the LLM's response generation accordingly. We extend DyPlan to DyPlan-verify, adding an internal verification and correction process to further enrich the generated answer. Experiments on three prominent multi-hop question answering (MHQA) datasets reveal how DyPlan can improve model performance by 7-13% while reducing the cost by 11-32% relative to the best baseline model.

Related papers

• LLM-Guided Evolutionary Search for Constraint Model Reformulation to Improve Solver Efficiency [5.774119507201688]
  Large Language Models (LLMs) can produce correct models from natural language, but even a correct model can be expensive to solve.<n>We use an evolutionary framework in which an LLM proposes candidate reformulations that are verified and benchmarked against the user-defined baseline model.<n>Existing retention strategies prioritize recency or performance, but do not explicitly diversify the context.<n>We systematically evaluate the strategies on eight CSPLib problems using validation-based final model selection.
  arXiv  Detail & Related papers  (2026-07-30T14:21:55Z)
• PaT: Planning-after-Trial for Efficient Test-Time Code Generation [25.264318601226265]
  Planning-after-Trial (PaT) is an adaptive policy for code generation that invokes a planner only upon verification failure.<n>We propose PaT, an adaptive policy for code generation that invokes a planner only upon verification failure.
  arXiv  Detail & Related papers  (2026-05-08T05:09:18Z)
• PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving [66.42260489147617]
  We introduce PLAN-TUNING, a framework that distills synthetic task decompositions from large-scale language models.<n>Plan-TUNING fine-tunes smaller models via supervised and reinforcement-learning objectives to improve complex reasoning.<n>Our analysis demonstrates how planning trajectories improves complex reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-10T07:30:44Z)
• SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science [1.1343849658875087]
  Large Language Models (LLMs) have revolutionized automated data analytics and machine learning by enabling dynamic reasoning and adaptability. We propose SPIO, a novel framework that orchestrates multi-agent planning across four key modules. In each module, dedicated planning agents independently generate candidate strategies that cascade into subsequent stages, fostering comprehensive exploration.
  arXiv  Detail & Related papers  (2025-03-30T04:45:32Z)
• Unveiling the Potential of Multimodal Retrieval Augmented Generation with Planning [5.205803766626321]
  Multimodal Retrieval Augmented Generation (MRAG) systems often rely on rigid, single-step retrieval methods. We present CogPlanner, a versatile framework inspired by human cognitive processes. CogPlanner iteratively refines queries and selects retrieval strategies, enabling both parallel and sequential modeling approaches.
  arXiv  Detail & Related papers  (2025-01-26T10:16:42Z)
• Interactive and Expressive Code-Augmented Planning with Large Language Models [62.799579304821826]
  Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. We propose REPL-Plan, an LLM planning approach that is fully code-expressive and dynamic.
  arXiv  Detail & Related papers  (2024-11-21T04:23:17Z)
• Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs [0.6700983301090584]
  Large language models (LLMs) are applied to more use cases, creating high quality, task-specific datasets for fine-tuning. Using high quality human data has been the most common approach to unlock model performance, but is prohibitively expensive in many scenarios. Several alternative methods have also emerged, such as generating synthetic or hybrid data, but the effectiveness of these approaches remain unclear.
  arXiv  Detail & Related papers  (2024-09-29T20:14:50Z)
• Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs [59.76268575344119]
  We introduce a novel framework for enhancing large language models' (LLMs) planning capabilities by using planning data derived from knowledge graphs (KGs) LLMs fine-tuned with KG data have improved planning capabilities, better equipping them to handle complex QA tasks that involve retrieval.
  arXiv  Detail & Related papers  (2024-06-20T13:07:38Z)
• Exploring and Benchmarking the Planning Capabilities of Large Language Models [57.23454975238014]
  This work lays the foundations for improving planning capabilities of large language models (LLMs) We construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. We investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance.
  arXiv  Detail & Related papers  (2024-06-18T22:57:06Z)
• Unlocking Large Language Model's Planning Capabilities with Maximum Diversity Fine-tuning [10.704716790096498]
  Large language models (LLMs) have demonstrated impressive task-solving capabilities, achieved through either prompting techniques or system designs. This paper investigates the impact of fine-tuning on LLMs' planning capabilities. We propose the Maximum Diversity Fine-Tuning (MDFT) strategy to improve the sample efficiency of fine-tuning in the planning domain.
  arXiv  Detail & Related papers  (2024-06-15T03:06:14Z)
• Guiding Language Model Reasoning with Planning Tokens [122.43639723387516]
  Large language models (LLMs) have recently attracted considerable interest for their ability to perform complex reasoning tasks. We propose a hierarchical generation scheme to encourage a more structural generation of chain-of-thought steps. Our approach requires a negligible increase in trainable parameters (0.001%) and can be applied through either full fine-tuning or a more parameter-efficient scheme.
  arXiv  Detail & Related papers  (2023-10-09T13:29:37Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.