When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

• URL: http://arxiv.org/abs/2606.23937v1
• Date: Mon, 22 Jun 2026 20:57:11 GMT
• Title: When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents
• Abstract summary: Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model.<n>We test this proxy for pre-action policy classification in tau-bench using Qwen2.5-3B/7B classifiers.
• Score: 13.484205922478163
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model. We test this proxy for pre-action policy classification in tau-bench using Qwen2.5-3B/7B classifiers. Under gold-policy conditioning, a compact structured state improves macro-F1 over raw trajectories by 0.13-0.17 after tuning. We then replace the benchmark-designated policy clause with the top-ranked clause retrieved from decision-time context. Although the exact governing clause is retrieved at rank 1 for only 7% of airline states, the primary 3B classifier obtains macro-F1 0.58 with retrieved clauses versus 0.60 with gold clauses (Delta=-0.02, task-cluster 95% CI [-0.23,+0.21]); mismatched-policy and no-policy controls score 0.32 and 0.21. We do not detect a macro-F1 difference between retrieved and gold clauses in this configuration, although the interval remains too wide to establish non-inferiority. The same qualitative pattern appears with a second retriever and at 7B, while varying across fine-tuning configurations. These results indicate that exact-match clause recall can underestimate downstream policy utility in this benchmark setting, motivating evaluation with retrieved policies in the classification loop rather than recall alone.

Related papers

• The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards [1.667501390377158]
  Group-relative policy optimization can discard nocontrast rollout groups through dynamic sampling.<n>We identify and quantify a metricpredicate mismatch under composite shaped rewards.
  arXiv  Detail & Related papers  (2026-09-12T10:36:55Z)
• Efficient GPU Retrieval for Semantic Search [11.631441668563802]
  We present a policy-aligned retrieval framework for LinkedIn.<n>A lightweight single-slot Stage-1 scorer generates high-recall candidates.<n>On 21K held-out queries, this representation improves offline relevance over a matched-capacity baseline.
  arXiv  Detail & Related papers  (2026-08-29T00:38:27Z)
• When Should Multi-Round RAG Stop? Structured Stopping Judgments and Retrieval Reduction in Search-R1 [0.0]
  Multi-round retrieval-augmented generation (RAG) must decide when to stop searching as evidence accumulates.<n>We adapt S2G-RAG's structured sufficiency-and-gap judgment to a frozen Search-R1 pipeline.<n>We train a Qwen3.5-2B judge on 3,009 states from 900 disjoint HotpotQA questions.
  arXiv  Detail & Related papers  (2026-08-13T13:42:48Z)
• Decision-Aware Memory Cards: Counterfactual-Inspired Context Selection and Compression for Tool-Using LLM Agents [3.964533007623828]
  Modern large language model (LLM) agents need decision-relevant evidence at the moment of action.<n>We present the Counterfactual-Inspired Context Layer (CICL), which builds an instance context graph, estimates decision-oriented utility for candidate units, and compresses selected evidence into typed memory cards.<n>CICL provides a practical layer for measuring, ranking, and compressing decision-critical context for tool-using agents.
  arXiv  Detail & Related papers  (2026-06-06T13:02:28Z)
• Benchmarking Recursive-Collapse Warning Claims Under Matched False-Positive Control [0.0]
  Recursive systems can enter collapse-like regimes before overt failure becomes visible.<n>We introduce Loopzero, a claim-bounded benchmark framework for testing whether failures follow a directional telemetry pattern.<n>We evaluate the bridge on two frozen public-artifact benchmarks.
  arXiv  Detail & Related papers  (2026-05-29T20:12:42Z)
• ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks [0.0]
  ATANT v1.0 defined continuity as a system property with 7 required properties.<n>We show by structural analysis that none of these benchmarks measures continuity as defined in v1.0.<n>We publish our 8.8% LOCOMO score alongside our 96% ATANT cumulative-scale score as a calibration pair.
  arXiv  Detail & Related papers  (2026-04-13T04:40:37Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Achieve Performatively Optimal Policy for Performative Reinforcement Learning [55.983627302691424]
  This work proposes a zeroth-order FrankWolfe- (0FW) algorithm with a gradient of performative policy in the framework.<n> Experimental results demonstrate that our 0FW is more effective than the existing approximation in finding the desired PO policy.
  arXiv  Detail & Related papers  (2025-10-06T01:56:31Z)
• Best-Effort Policies for Robust Markov Decision Processes [69.60742680559788]
  We study the common generalization of Markov decision processes (MDPs) with sets of transition probabilities, known as robust MDPs (RMDPs)<n>We call such a policy an optimal robust best-effort (ORBE) policy.<n>We prove that ORBE policies always exist, characterize their structure, and present an algorithm to compute them with a small overhead compared to standard robust value iteration.
  arXiv  Detail & Related papers  (2025-08-11T09:18:34Z)
• Code Compliance Assessment as a Learning Problem [0.15229257192293197]
  We formulate code compliance assessment as a machine learning (ML) problem. We generate a prediction on the code's compliance, non-compliance, or irrelevance. Our tool Policy2Code achieves classification accuracies of (59%, 71%) and search MRR of (0.05, 0.21) In a user study, 24% Policy2Code detections were accepted compared to 7% for CodeBERT.
  arXiv  Detail & Related papers  (2022-09-10T05:41:04Z)
• Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking [57.42241521034744]
  We propose the concept of certified error control of candidate set pruning for relevance ranking. Our method successfully prunes the first-stage retrieved candidate sets to improve the second-stage reranking speed.
  arXiv  Detail & Related papers  (2022-05-19T16:00:13Z)
• Supervised Off-Policy Ranking [145.3039527243585]
  Off-policy evaluation (OPE) leverages data generated by other policies to evaluate a target policy. We propose supervised off-policy ranking that learns a policy scoring model by correctly ranking training policies with known performance. Our method outperforms strong baseline OPE methods in terms of both rank correlation and performance gap between the truly best and the best of the ranked top three policies.
  arXiv  Detail & Related papers  (2021-07-03T07:01:23Z)
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details [107.2722027807328]
  We find that the default implementation of AP is neither category independent, nor does it directly reward properly calibrated detectors. We show that the default implementation produces a gameable metric, where a simple, nonsensical re-ranking policy can improve AP by a large margin. We benchmark recent advances in large-vocabulary detection and find that many reported gains do not translate to improvements under our new per-class independent evaluation.
  arXiv  Detail & Related papers  (2021-02-01T18:56:02Z)
• Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies [80.42316902296832]
  We study the estimation of policy value and gradient of a deterministic policy from off-policy data when actions are continuous. In this setting, standard importance sampling and doubly robust estimators for policy value and gradient fail because the density ratio does not exist. We propose several new doubly robust estimators based on different kernelization approaches.
  arXiv  Detail & Related papers  (2020-06-06T15:52:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.