Efficient Test-Time Adaptation through Human-AI Interaction
- URL: http://arxiv.org/abs/2609.04141v1
- Date: Thu, 03 Sep 2026 17:33:18 GMT
- Title: Efficient Test-Time Adaptation through Human-AI Interaction
- Abstract summary: We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
- Score: 112.5131603022122
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: AI agents are trained on population-scale data to encode broad capabilities spanning those of many practitioners. Yet the artifacts they produce rarely meet the personal bar professionals need to stake their reputation on. On realistic, open-ended tasks where success criteria are heterogeneous and insufficiently documented, individual expertise lives precisely in the elevation and departure from the average. In practice, iterative human-agent interaction surfaces criteria that users cannot fully specify up front, yet apply repeatedly across tasks. We argue this cross-session interaction data is a rich, underused signal for closing the gap to individual expertise. In this work, we propose test-time adaptation through human-agent interaction (TAHI), which integrates these signals into agent context and weights, and crystallizes each user's training and evaluation criteria via an evolving rubric module. We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks. Our agents improve solo task success by 4.5-20.9% within only tens of tasks. Meanwhile, our evolving rubric module serves as a scalable annotation tool, creating evaluation rubrics that catch 16.0-22.3% more failures than those from LMs or humans alone. While agents are adapted towards individuals, we show these personalized agents also produce improvements in success of up to 8.8% that generalize across users.
Related papers
- EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents [75.01735520608075]
Existing benchmarks mostly test literal Theory of Mind (ToM) by asking direct belief questions.<n>We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household.<n>On the hard split, all seven evaluated frontier models score 0.0% Pass3 on functional task completion, while averaging 45.0% on literal belief probes.
arXiv Detail & Related papers (2026-05-11T00:04:19Z) - Modeling Distinct Human Interaction in Web Agents [59.600507469754575]
We introduce the task of modeling human intervention to support collaborative web task execution.<n>We identify four distinct patterns of user interaction with agents -- hands-off supervision, hands-on oversight, collaborative task-solving, and full user takeover.<n>We deploy these intervention-aware models in live web navigation agents and evaluate them in a user study, finding a 26.5% increase in user-rated agent usefulness.
arXiv Detail & Related papers (2026-02-19T18:11:28Z) - Autonomous Continual Learning of Computer-Use Agents for Environment Adaptation [57.65688895630163]
We introduce ACuRL, an Autonomous Curriculum Reinforcement Learning framework that continually adapts agents to specific environments with zero human data.<n>Our method effectively enables both intra-environment and cross-environment continual learning, yielding 4-22% performance gains without forgetting existing environments.
arXiv Detail & Related papers (2026-02-10T23:06:02Z) - AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
arXiv Detail & Related papers (2026-01-28T13:49:18Z) - Training LLM Agents to Empower Humans [67.80021254324294]
We propose a new approach to tuning assistive language models based on maximizing the human's empowerment.<n>Our empowerment-maximizing method, Empower, only requires offline text data.<n>We show that agents trained with Empower increase the success rate of a simulated human programmer on challenging coding questions by an average of 192%.
arXiv Detail & Related papers (2025-10-15T16:09:33Z) - PersonaGym: Evaluating Persona Agents and LLMs [44.058616871217595]
We introduce PersonaGym, the first dynamic evaluation framework for persona agents, and PersonaScore, a human-aligned automatic metric grounded in decision theory.<n>Our evaluation of 10 leading LLMs across 200 personas and 10,000 questions reveals significant advancement opportunities.
arXiv Detail & Related papers (2024-07-25T22:24:45Z) - MUG: Interactive Multimodal Grounding on User Interfaces [12.035123646959669]
We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen.
Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command. MUG allows multiple rounds of interactions such that upon seeing the agent responses, the user can give further commands for the agent to refine or even correct its actions.
arXiv Detail & Related papers (2022-09-29T21:08:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.