Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents

• URL: http://arxiv.org/abs/2605.06908v1
• Date: Thu, 07 May 2026 20:15:51 GMT
• Title: Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents
• Abstract summary: LLM agents aim to invoke extra computation only when it improves performance.<n>Existing methods typically use confidence-, uncertainty-, or difficulty-based gates, assuming a fixed direction from the gating signal through compute need to the value of computation.<n>We show that this alignment is unstable, with reversals across environments and backbones even when the task is fixed.<n>We propose DIAL (Direction-Informed Adaptive Learning), a sparse gate trained from signal-agnostic counterfactual exploration to learn the utility direction of state features per.
• Score: 24.71914991820339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adaptive test-time compute for LLM agents aims to invoke extra computation only when it improves performance. Existing methods typically use confidence-, uncertainty-, or difficulty-based gates, assuming a fixed direction from the gating signal through compute need to the value of computation. This makes gating a utility-calibration problem: gating signals should align with whether extra computation improves the final outcome over the base policy. We show that this alignment is unstable: the same signal predicts rollout benefit in one setting and rollout harm in another, with reversals across environments and backbones even when the task is fixed. Wrong-direction gates can therefore worsen performance by precisely selecting harmful states. This reversal reflects a deeper distinction between compute need and compute suitability: a high uncertainty signal may indicate decision-difficult states where rollouts help compare alternatives, or intervention-unsuitable states where the current context does not support useful rollout-based improvement. Under this two-source model, fixed-direction gates are unreliable across heterogeneous settings. To address this, we propose DIAL (Direction-Informed Adaptive Learning), a sparse gate trained from signal-agnostic counterfactual exploration to learn the utility direction of state features per (environment, backbone). Across six environments and three backbones, DIAL yields a stronger overall success-cost trade-off than fixed-direction baselines.

Related papers

• Relational Over-Regularization: Graph-Based AI-Generated Text Detection via Sentence Transition Deviation [4.4331439696271415]
  CrossSource Stylometric Fingerprint (CSFG) is a graph-based framework that encodes positional, sequential, semantic and transition deviation signals as learnable GNN edge features.<n>CSFG achieves 97.14% accuracy under binary detection, outperforming the strongest graph-based baseline by 11.14 pp.
  arXiv  Detail & Related papers  (2026-08-27T06:51:40Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning [25.144030941321237]
  Geoalign is a plug-in for rollout curation in iterative policy optimization.<n>It forms within-prompt preference pairs, learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and rectifies them with within-prompt stable alternatives.<n>Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards, Geoalign improves final performance and reduces training oscillation.
  arXiv  Detail & Related papers  (2026-06-25T11:53:37Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling [40.94400211806987]
  We propose a policy-driven ZO framework that treats the sampling distribution over perturbation directions as a learnable policy.<n>We show that learned sampling improves quality gradient information and relax the explicit dependence on $d$ in convergence bounds.<n>Our results suggest that adaptive direction sampling is a promising route to make ZO fine-tuning viable at scale.
  arXiv  Detail & Related papers  (2026-02-14T08:01:41Z)
• Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking [25.69863022367215]
  Reasoning prompting can enhance ranking utility, but its benefits are inconsistent and come at a substantial computational cost.<n>We propose a reasoning routing framework that employs a lightweight, plug-and-play router head to decide whether to use direct inference (Non-Think) or reasoning (Think) for each instance before generation.
  arXiv  Detail & Related papers  (2026-01-26T05:09:07Z)
• Learning to Ponder: Adaptive Reasoning in Latent Space [2.8835557003761747]
  We present FR-Ponder, a single-graph, backbone-training-free framework that allocates instance-adaptive reasoning compute via latent steering.<n>On GSM8K and MATH500, FR-Ponder improves the compute-accuracy frontier, delivering lower FLOPs with better matched accuracy and comparing favorably to early-exit baselines.
  arXiv  Detail & Related papers  (2025-09-29T03:21:42Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling [84.00480999255628]
  Reinforcement Learning algorithms for safety alignment of Large Language Models (LLMs) encounter the challenge of distribution shift.<n>Current approaches typically address this issue through online sampling from the target policy.<n>We propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals.
  arXiv  Detail & Related papers  (2025-03-13T06:40:34Z)
• Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization [61.02719787737867]
  Large language models (LLMs) are increasingly deployed and democratized on edge devices.<n>One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM.<n>We conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings.
  arXiv  Detail & Related papers  (2025-02-06T18:59:11Z)
• Reinforcement learning with non-ergodic reward increments: robustness via ergodicity transformations [8.44491527275706]
  Application areas for reinforcement learning include autonomous driving, precision agriculture, and finance.<n>In particular, the focus of RL is typically on the expected value of the return.<n>We develop an algorithm that lets RL agents optimize the long-term performance of individual trajectories.
  arXiv  Detail & Related papers  (2023-10-17T15:13:33Z)
• Adaptive Self-supervision Algorithms for Physics-informed Neural Networks [59.822151945132525]
  Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function. We study the impact of the location of the collocation points on the trainability of these models. We propose a novel adaptive collocation scheme which progressively allocates more collocation points to areas where the model is making higher errors.
  arXiv  Detail & Related papers  (2022-07-08T18:17:06Z)
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction [96.90215318875859]
  We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from corrective feedback. We propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training.
  arXiv  Detail & Related papers  (2020-03-16T16:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.