HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation

• URL: http://arxiv.org/abs/2607.04329v1
• Date: Sun, 05 Jul 2026 14:23:27 GMT
• Title: HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation
• Abstract summary: We introduce a graph-based framework that represents humans and agents as first-class participants with explicit roles, permissions, communication paths, and action authority.<n>Building on this framework, HAS-Bench evaluates Human-Agent Systems under human participation across agency levels, interaction channels, and persona policies.<n> Experiments across six domains show that human participation can substantially improve task completion and failure recovery, but the gains depend on when, how, and by whom human input is exercised.
• Score: 54.200005792928344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models increasingly operate in settings where humans are active collaborators rather than passive task providers. We introduce HAS-Framework, a graph-based framework that represents humans and LLM-powered agents as first-class participants with explicit roles, permissions, communication paths, and action authority. Building on this framework, HAS-Bench evaluates Human-Agent Systems under configurable human participation across agency levels, interaction channels, and persona policies. The benchmark measures both task outcomes and process-level collaboration behavior, including clarification quality, feedback utilization, control calibration, safety, initiative, and interaction cost. Experiments across six domains show that human participation can substantially improve task completion and failure recovery, but the gains depend on when, how, and by whom human input is exercised.

Related papers

• LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior [51.25229110468212]
  Embodied agents operating in decentralized environments often exhibit behaviors that are misaligned with their partners or inconsistent with the environment state.<n>We propose a novel framework, Learning Laws of Cooperation, that enables embodied agents to autonomously align with both their partners and task objectives.<n>Our framework allows agents to reflect on past failures to extract misaligned behavioral patterns, which are used to derive high-level behavioral laws, such as "Talk when necessary" and "Wait for partner"
  arXiv  Detail & Related papers  (2026-06-26T15:26:12Z)
• CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments [32.043112677406064]
  CollabSim is a simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of agents' internal states.<n> Experiments show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of agent design.
  arXiv  Detail & Related papers  (2026-06-04T17:06:22Z)
• Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration [54.18136393879849]
  We present ALMANAC, a dataset of action-level mental model ANnotations for Agent Collaboration.<n>ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations.<n>Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors.
  arXiv  Detail & Related papers  (2026-06-04T16:56:12Z)
• SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy [59.86199649493784]
  Controllable cooperative humanoid manipulation is a fundamental yet challenging problem for embodied intelligence.<n>We present SynAgent, a framework that enables scalable and physically plausible cooperative manipulation.<n>In experiments, SynAgent significantly outperforms existing baselines in both cooperative imitation and trajectory-conditioned control.
  arXiv  Detail & Related papers  (2026-04-20T17:46:20Z)
• LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey [34.275920463375684]
  Large language models (LLMs) have sparked growing interest in building fully autonomous agents.<n>LLM-HAS incorporate human-provided information, feedback, or control into the agent system to enhance system performance, reliability and safety.<n>This paper provides the first comprehensive and structured survey of LLM-HAS.
  arXiv  Detail & Related papers  (2025-05-01T08:29:26Z)
• VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures [3.075266204492352]
  Large language model (LLM) agents in compound AI systems often fail to meet human standards, leading to errors that compromise the system's overall performance.<n>This paper introduces a human-centered evaluation framework for verifying LLM Agent failures (VeriLA)<n>VeriLA systematically assesses agent failures to reduce human effort and make these agent failures interpretable to humans.
  arXiv  Detail & Related papers  (2025-03-16T21:11:18Z)
• Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration [50.657070334404835]
  Collaborative Gym is a framework enabling asynchronous, tripartite interaction among agents, humans, and task environments.<n>We instantiate Co-Gym with three representative tasks in both simulated and real-world conditions.<n>Our findings reveal that collaborative agents consistently outperform their fully autonomous counterparts in task performance.
  arXiv  Detail & Related papers  (2024-12-20T09:21:15Z)
• Large Language Model-based Human-Agent Collaboration for Complex Task Solving [94.3914058341565]
  We introduce the problem of Large Language Models (LLMs)-based human-agent collaboration for complex task-solving. We propose a Reinforcement Learning-based Human-Agent Collaboration method, ReHAC. This approach includes a policy model designed to determine the most opportune stages for human intervention within the task-solving process.
  arXiv  Detail & Related papers  (2024-02-20T11:03:36Z)
• MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming [27.911816995891726]
  We introduce MetaAgents, a social simulation framework populated with Large Language Models (LLMs)<n>We construct a job fair environment as a case study to scrutinize the team assembly and skill-matching behaviors of LLM-based agents.<n>Our evaluation demonstrates that LLM-based agents perform competently in making rational decisions to develop efficient teams.
  arXiv  Detail & Related papers  (2023-10-10T10:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.