MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking

• URL: http://arxiv.org/abs/2607.17751v2
• Date: Wed, 29 Jul 2026 09:51:16 GMT
• Title: MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking
• Abstract summary: MagicSelector is a framework capable of translating ambiguous user instructions into executable atomic subtasks and guiding high-precision tool retrieval.<n>We empower MagicSelector with capabilities through three key contributions: (1) a preference-guided counterfactual task decomposition mechanism; (2) a progressive tool reranking method driven by self-distillation hard negative mining; and (3) a dual semantic boundary-aware dynamic Top-K strategy.<n>Experiments demonstrate that MagicSelector significantly outperforms state-of-the-art methods in terms of tool retrieval accuracy, OOD generalization capability, and overall token efficiency.
• Score: 31.430385887150972
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present MagicSelector, a joint optimization framework integrating Counterfactual task decomposition, Progressive reranking, and Dynamic Top-K, designed to address the fundamental challenges of tool retrieval in agents. MagicSelector is a specialized framework capable of translating ambiguous user instructions into executable atomic subtasks and guiding high-precision tool retrieval, effectively mitigating redundant noise and severe context distraction in out-of-domain (OOD) scenarios. We empower MagicSelector with these capabilities through three key contributions: (1) a preference-guided counterfactual task decomposition mechanism that utilizes a counterfactual reward to quantify the marginal causal gain of decomposition on retrieval ranking, effectively imposing fine-grained structural supervision on logical coherence; (2) a progressive tool reranking method driven by self-distillation hard negative mining, which optimizes both point-wise and list-wise relevance to enhance fine-grained discrimination among highly similar tools; and (3) a dual semantic boundary-aware dynamic Top-K strategy that adaptively monitors reranking score cliffs and inter-tool semantic shifts to dynamically truncate the candidate list, maximizing relevant tool recall while filtering long-tail noise. Evaluated on MTDTool, the first task decomposition benchmark we constructed tailored for mobile multi-turn interactions with process-level annotations, MagicSelector yields promising performance. Extensive experiments demonstrate that MagicSelector significantly outperforms state-of-the-art methods in terms of tool retrieval accuracy, OOD generalization capability, and overall token efficiency, thereby demonstrating the effectiveness of our proposed framework.

Related papers

• PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval [32.19528642807244]
  Task decomposition aims to transform ambiguous instructions into executable atomic subtasks.<n>Rewards for task decomposition can easily induce reward hacking in reinforcement learning-based methods.<n>We propose PCTD, a Preference-guided Counterfactual Task Decomposition framework.
  arXiv  Detail & Related papers  (2026-07-17T07:15:41Z)
• From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents [90.3279207163486]
  EvoSOP is a framework that empowers agents to extract SOPs from execution trajectories and iteratively optimize the toolset.<n>We show that EvoSOP significantly boosts task success rates while substantially reducing the number of interaction rounds.
  arXiv  Detail & Related papers  (2026-07-08T12:09:49Z)
• Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models [27.250148827297604]
  HDPO is a framework that reframes tool efficiency from a competing scalar objective to a strictly conditional one.<n>Our resulting model, Metis, reduces tool invocations by orders of magnitude while simultaneously elevating reasoning accuracy.
  arXiv  Detail & Related papers  (2026-04-09T17:59:57Z)
• Adaptive Robust Estimator for Multi-Agent Reinforcement Learning [27.595086716369483]
  We propose a robust multi-agent reinforcement learning framework for collaborative reasoning.<n>It consists of two components: Dual-Agent Answer-Critique-Rewrite (DACR) and an Adaptive Robust Estimator (ARE)
  arXiv  Detail & Related papers  (2026-03-23T04:51:15Z)
• Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents [54.18201810286764]
  Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering.<n>In long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance.<n>We propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior.
  arXiv  Detail & Related papers  (2026-02-02T12:52:14Z)
• PerfGuard: A Performance-Aware Agent for Visual Content Generation [53.591105729011595]
  PerfGuard is a performance-aware agent framework for visual content generation.<n>It integrates tool performance boundaries into task planning and scheduling.<n>It has advantages in tool selection accuracy, execution reliability, and alignment with user intent.
  arXiv  Detail & Related papers  (2026-01-30T05:12:19Z)
• MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimization [52.149337961205624]
  We propose a framework that empowers both inter- and intra-task optimization for surgical triplet recognition.<n>For inter-task optimization, we introduce the Shared-Specific-Disentangled (S$2$D) learning scheme that decomposes representations into task-shared and task-specific components.<n>For intra-task optimization conflicts, we develop a Coordinated Gradient Learning (CGL) strategy, which dissects and rebalances the positive-negative ambiguities.
  arXiv  Detail & Related papers  (2025-09-16T09:48:52Z)
• Acting Less is Reasoning More! Teaching Model to Act Efficiently [87.28134636548705]
  Tool-integrated reasoning augments large language models with the ability to invoke external tools to solve tasks.<n>Current approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use.<n>We propose a framework that encourages models to produce accurate answers with minimal tool calls.<n>Our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.
  arXiv  Detail & Related papers  (2025-04-21T05:40:05Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• AdaRank: Adaptive Rank Pruning for Enhanced Model Merging [23.649762835129167]
  Model merging has emerged as a promising approach for unifying independently fine-tuned models into an integrated framework.<n>We propose AdaRank, a novel model merging framework that adaptively selects the most beneficial singular directions of task vectors to merge multiple models.<n>AdaRank consistently achieves state-of-the-art performance with various backbones and number of tasks, reducing the performance gap between fine-tuned models to nearly 1%.
  arXiv  Detail & Related papers  (2025-03-28T06:49:06Z)
• Meta-Wrapper: Differentiable Wrapping Operator for User Interest Selection in CTR Prediction [97.99938802797377]
  Click-through rate (CTR) prediction, whose goal is to predict the probability of the user to click on an item, has become increasingly significant in recommender systems. Recent deep learning models with the ability to automatically extract the user interest from his/her behaviors have achieved great success. We propose a novel approach under the framework of the wrapper method, which is named Meta-Wrapper.
  arXiv  Detail & Related papers  (2022-06-28T03:28:15Z)
• Uncertainty-Guided Efficient Interactive Refinement of Fetal Brain Segmentation from Stacks of MRI Slices [13.257420793113084]
  We propose an Uncertainty-Guided Interactive Refinement framework to improve the efficiency of interactive refinement process. A novel interactive level set method is also proposed to obtain a refined result given the initial segmentation and user interactions. Experimental results show that: (1) our proposed CNN obtains uncertainty estimation in real time which correlates well with mis-segmentations, (2) the proposed interactive level set is effective and efficient for refinement, and (3) UGIR obtains accurate refinement results with around 30% improvement of efficiency by using uncertainty to guide user interactions.
  arXiv  Detail & Related papers  (2020-07-02T01:50:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.