Learning to Learn-at-Test-Time: Language Agents with Learnable Adaptation Policies

• URL: http://arxiv.org/abs/2604.00830v2
• Date: Thu, 02 Apr 2026 09:44:30 GMT
• Title: Learning to Learn-at-Test-Time: Language Agents with Learnable Adaptation Policies
• Abstract summary: Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time.<n>We argue that optimal adaptation policies should be learned from task environments, not hand-engineered based on human intuition.<n>We introduce Meta-TTL, a framework that formulates the discovery of effective adaptation policies as a bi-level optimization problem.
• Score: 48.31565681022125
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is an adaptation policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods rely on fixed, hand-crafted adaptation policies rather than optimizing them for downstream improvement. We argue that optimal adaptation policies should be learned from task environments, not hand-engineered based on human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective adaptation policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate adaptation policy helps an agent correct errors across sequential episodes. Guided by the agent's performance, the outer loop employs evolutionary search over a diverse distribution of training tasks to iteratively refine the adaptation policy. We evaluate Meta-TTL on Jericho and WebArena-Lite across both in-distribution (ID) and out-of-distribution (OOD) settings, using multiple meta-agent backbones. Results on both benchmarks show that Meta-TTL consistently outperforms hand-crafted baselines, suggesting that the optimized adaptation policy encodes transferable strategies that generalize beyond the training task distribution.

Related papers

• TAPO: Transition-Aware Policy Optimization for LLM Agents [27.95488836373643]
  TAPO is a unified training framework that alternates between policy optimization and transition supervision.<n>It serves as a computationally lightweight, plug-and-play enhancement module for existing agent RL algorithms.<n>We conduct systematic experiments on WebShop and ALFWorld, integrating foundation models of various scales with different policy optimization algorithms.
  arXiv  Detail & Related papers  (2026-07-30T10:17:55Z)
• PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning [8.197606162420145]
  We propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold.<n>Our framework, PATS, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts.
  arXiv  Detail & Related papers  (2026-07-23T15:24:35Z)
• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning [38.54413500261524]
  Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions.<n>We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills.
  arXiv  Detail & Related papers  (2026-07-16T09:57:18Z)
• PAWS: Preference Learning with Advantage-Weighted Segments [61.590521943273]
  We propose a segment-based preference learning method that performs policy updates directly using segment-level advantage functions.<n>Experiments on simulated robotic manipulation and locomotion tasks demonstrate that PAWS consistently outperforms existing PbRL approaches.
  arXiv  Detail & Related papers  (2026-06-10T12:00:17Z)
• TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models [49.463896453707065]
  Vision-Language-Action (VLA) models have made remarkable progress, but they remain vulnerable to distribution shifts at deployment time.<n>Recent VLA models suggest that prompts can serve as an efficient interface for steering policy behavior, but existing prompt-based steering typically relies on external guidance.<n>This raises a natural question: can test-time training (TTT) for VLA be achieved by optimizing a prompt, so that the steering interface itself can be learned and adapted from interaction?<n>We address this question with TTT-VLA, a test-time training framework based on Latent Prompt Optimization (LPO)
  arXiv  Detail & Related papers  (2026-06-02T04:10:39Z)
• Align and Filter: Improving Performance in Asynchronous On-Policy RL [27.989398323927393]
  We identify the sources of policy lag caused by distributed learning and high update frequency.<n>We propose textittotal Variation-based Advantage aligned Constrained policy Optimization as a practical approach to mitigate policy lag.
  arXiv  Detail & Related papers  (2026-03-02T01:52:34Z)
• In-Context Reinforcement Learning From Suboptimal Historical Data [56.60512975858003]
  Transformer models have achieved remarkable empirical successes, largely due to their in-context learning capabilities.<n>We propose the Decision Importance Transformer framework, which emulates the actor-critic algorithm in an in-context manner.<n>Our results show that DIT achieves superior performance, particularly when the offline dataset contains suboptimal historical data.
  arXiv  Detail & Related papers  (2026-01-27T23:13:06Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Policy-Conditioned Policies for Multi-Agent Task Solving [53.67744322553693]
  In this work, we propose a paradigm shift that bridges the gap by representing policies as human-interpretable source code.<n>We reformulate the learning problem by utilizing Large Language Models (LLMs) as approximate interpreters.<n>We formalize this process as textitProgrammatic Iterated Best Response (PIBR), an algorithm where the policy code is optimized by textual gradients.
  arXiv  Detail & Related papers  (2025-12-24T07:42:10Z)
• TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization [12.061547251822326]
  Trajectory-based Group Relative Policy Optimization (TGRPO) is an online RL-based training framework for Visual-Language-Action (VLA) models.<n>We show that TGRPO achieves an average success rate of 80.7%, which is 4.2% higher than that of Supervised Fine-Tuning (SFT) and outperforms other representative RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-06-10T04:27:49Z)
• Adaptive Augmentation Policy Optimization with LLM Feedback [3.038642416291856]
  Data augmentation is a critical component of deep learning pipelines, enhancing model generalization by increasing dataset diversity.<n>Traditional augmentation strategies rely on manually designed transformations, classification sampling, or automated search-based approaches.<n>We propose a Large Language Model (LLM)-guided augmentation optimization strategy that refines augmentation policies based on model performance feedback.
  arXiv  Detail & Related papers  (2024-10-17T11:26:10Z)
• Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator [9.900800253949512]
  We develop a bilevel optimization framework for meta-RL (BO-MRL) to learn the meta-prior for task-specific policy adaptation. We empirically validate the correctness of the derived upper bounds and demonstrate the superior effectiveness of the proposed algorithm over benchmarks.
  arXiv  Detail & Related papers  (2024-10-13T05:17:58Z)
• Towards Efficient Exact Optimization of Language Model Alignment [93.39181634597877]
  Direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. We show that DPO derived based on the optimal solution of problem leads to a compromised mean-seeking approximation of the optimal solution in practice. We propose efficient exact optimization (EXO) of the alignment objective.
  arXiv  Detail & Related papers  (2024-02-01T18:51:54Z)
• Acceleration in Policy Optimization [50.323182853069184]
  We work towards a unifying paradigm for accelerating policy optimization methods in reinforcement learning (RL) by integrating foresight in the policy improvement step via optimistic and adaptive updates. We define optimism as predictive modelling of the future behavior of a policy, and adaptivity as taking immediate and anticipatory corrective actions to mitigate errors from overshooting predictions or delayed responses to change. We design an optimistic policy gradient algorithm, adaptive via meta-gradient learning, and empirically highlight several design choices pertaining to acceleration, in an illustrative task.
  arXiv  Detail & Related papers  (2023-06-18T15:50:57Z)
• Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation [17.165083095799712]
  We study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback.
  arXiv  Detail & Related papers  (2022-11-20T03:55:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.