StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

• URL: http://arxiv.org/abs/2608.23475v1
• Date: Mon, 24 Aug 2026 16:41:10 GMT
• Title: StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models
• Abstract summary: Few-shot in-context learning (ICL) has become a key paradigm for task adaptation.<n>Direct ICL often uses a small set of examples without explicitly abstracting task rules.<n>Human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances.
• Score: 36.381926644656765
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.

Related papers

• Strategy-Induct: Task-Level Strategy Induction for Instruction Generation [23.56553381765231]
  Strategy-Induct is a framework that derives task-level instructions solely from a small set of example questions.<n>Our approach first prompts the model to generate explicit reasoning strategies for each question, forming (strategy, question) pairs.<n>These pairs are then used to induce a task instruction that guides reasoning.
  arXiv  Detail & Related papers  (2026-05-20T09:10:43Z)
• Automated Instruction Revision (AIR): A Structured Comparison of Task Adaptation Strategies for LLM [0.0]
  This paper studies Automated Instruction Revision (AIR), a rule-induction-based method for adapting large language models to downstream tasks.<n>We position AIR within the broader landscape of adaptation strategies, including prompt optimization, retrieval-based methods, and fine-tuning.<n>We compare these approaches across a diverse benchmark suite designed to stress different task requirements, such as knowledge injection, structured extraction, label remapping, and logical reasoning.
  arXiv  Detail & Related papers  (2026-04-10T15:28:05Z)
• Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance [86.46794021499511]
  We show a previously underexplored gap between strategy usage and strategy executability.<n>We propose Selective Strategy Retrieval (SSR), a test-time framework that explicitly models executability.<n> SSR yields reliable and consistent improvements over direct solving, in-context learning, and single-source guidance.
  arXiv  Detail & Related papers  (2026-02-26T03:34:23Z)
• Experience-Guided Adaptation of Inference-Time Reasoning Strategies [49.954515048847874]
  Experience-Guided Reasoner (EGuR) generates tailored strategies at inference time based on accumulated experience.<n>EGuR achieves up to 14% accuracy improvements over the strongest baselines while reducing computational costs by up to 111x.
  arXiv  Detail & Related papers  (2025-11-14T17:45:28Z)
• Plan before Solving: Problem-Aware Strategy Routing for Mathematical Reasoning with LLMs [49.995906301946]
  Existing methods usually leverage a fixed strategy to guide Large Language Models (LLMs) to perform mathematical reasoning.<n>Our analysis reveals that the single strategy cannot adapt to problem-specific requirements and thus overlooks the trade-off between effectiveness and efficiency.<n>We propose Planning and Routing through Instance-Specific Modeling (PRISM), a novel framework that decouples mathematical reasoning into two stages: strategy planning and targeted execution.
  arXiv  Detail & Related papers  (2025-09-29T07:22:41Z)
• STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment [24.80531387685099]
  We propose a novel two-stage exemplar selection strategy that achieves a strong balance between efficiency, generalizability, and performance.<n>First, we fine-tune a BERT-based retriever using structure-aware supervision, guiding it to select exemplars that are both semantically relevant and structurally aligned.<n>Then, we enhance the retriever with a plug-in module, which amplifies syntactically meaningful information in the hidden representations.
  arXiv  Detail & Related papers  (2025-08-28T16:04:39Z)
• StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving [76.5322280307861]
  StrategyLLM allows LLMs to perform inductive reasoning, deriving general strategies from specific task instances, and deductive reasoning, applying these general strategies to particular task examples, for constructing generalizable and consistent few-shot prompts. Experimental results demonstrate that StrategyLLM outperforms the competitive baseline CoT-SC that requires human-annotated solutions on 13 datasets across 4 challenging tasks without human involvement, including math reasoning (34.2% $rightarrow$ 38.8%), commonsense reasoning (70.3% $rightarrow$ 72.5%), algorithmic reasoning (73.7% $rightarrow$ 85.0
  arXiv  Detail & Related papers  (2023-11-15T09:18:09Z)
• In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax [36.98247762224868]
  In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks. Do models infer the underlying structure of the task defined by the context, or do they rely on superficial generalizations that only generalize to identically distributed examples? In experiments with models from the GPT, PaLM, and Llama 2 families, we find large variance across LMs. The variance is explained more by the composition of the pre-training corpus and supervision methods than by model size.
  arXiv  Detail & Related papers  (2023-11-13T23:52:43Z)
• Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction [67.54420015049732]
  Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. Existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings.
  arXiv  Detail & Related papers  (2023-05-23T18:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.