CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

• URL: http://arxiv.org/abs/2606.09833v1
• Date: Mon, 20 Apr 2026 03:03:25 GMT
• Title: CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks
• Abstract summary: CollabSkill is a framework for evaluating human-agent collaboration on real-world occupational tasks.<n>It pairs real human workers with AI agents on tasks matched to their occupational background.<n>Our analysis reveals that practical experience emerges as the primary driver of collaboration skill.
• Score: 55.84031315625504
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: AI agents are reshaping the workspace, leading to drastic change of how humans work. Despite the considerable potential of human-agent collaboration both in preserving human agency and generating economic value, this paradigm remains largely absent from occupational task evaluation, hindered by the difficulty of gathering real human data and accounting for inter-human variability. We introduce CollabSkill, a framework for evaluating human-agent collaboration on real-world occupational tasks. CollabSkill pairs real human workers with AI agents on tasks matched to their occupational background, collecting data that capture the complexity of economically valuable tasks and the usage patterns of real workers. To account for inter-human variability, CollabSkill employs a Bayesian skill rating system to disentangle and quantify the skill contributions of both humans and AI agents. Drawing on over 1,500 prompts from 386 working sessions contributed by 93 human workers, our analysis yields insights on two fronts: on the agent side, rankings on CollabSkill diverge meaningfully from those of existing fully autonomous benchmarks where Codex leads, with Claude Code ranking first; on the human side, CollabSkill reveals that practical experience emerges as the primary driver of collaboration skill, with hands-on collaboration meaningfully shifting workers' AI literacy. Together, we hope CollabSkill enables the community to invest in systematic evaluation of human-agent collaboration and spurs development efforts aimed at building AI agents that genuinely augment human workers.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration [54.18136393879849]
  We present ALMANAC, a dataset of action-level mental model ANnotations for Agent Collaboration.<n>ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations.<n>Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors.
  arXiv  Detail & Related papers  (2026-06-04T16:56:12Z)
• How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations [112.57167042285437]
  We study how agents do human work by presenting the first direct comparison of human and agent workers.<n>We find that agents deliver results 88.3% faster and cost 90.4-96.2% less than humans.
  arXiv  Detail & Related papers  (2025-10-26T18:10:22Z)
• Towards Effective Human-in-the-Loop Assistive AI Agents [15.11527529177358]
  We introduce an evaluation framework and a dataset of human-AI interactions to assess how AI guidance affects procedural task performance.<n>We also develop an AR-equipped AI agent that provides interactive guidance in real-world tasks, from cooking to battlefield medicine.
  arXiv  Detail & Related papers  (2025-07-24T12:50:46Z)
• A Task-Driven Human-AI Collaboration: When to Automate, When to Collaborate, When to Challenge [16.734679201317896]
  We show how proper human-AI integration maintains meaningful agency while improving performance.<n>This framework lays the foundation for practically effective and morally sound human-AI collaboration.
  arXiv  Detail & Related papers  (2025-05-23T23:19:15Z)
• Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration [50.657070334404835]
  Collaborative Gym is a framework enabling asynchronous, tripartite interaction among agents, humans, and task environments.<n>We instantiate Co-Gym with three representative tasks in both simulated and real-world conditions.<n>Our findings reveal that collaborative agents consistently outperform their fully autonomous counterparts in task performance.
  arXiv  Detail & Related papers  (2024-12-20T09:21:15Z)
• Efficient Human-AI Coordination via Preparatory Language-based Convention [17.840956842806975]
  Existing methods for human-AI coordination typically train an agent to coordinate with a diverse set of policies or with human models fitted from real human data. We propose employing the large language model (LLM) to develop an action plan that effectively guides both human and AI. Our method achieves better alignment with human preferences and an average performance improvement of 15% compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-11-01T10:18:23Z)
• Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction [0.0]
  We show that task performance and task satisfaction improve through AI delegation. We identify humans' increased levels of self-efficacy as the underlying mechanism for these improvements. Our findings provide initial evidence that allowing AI models to take over more management responsibilities can be an effective form of human-AI collaboration.
  arXiv  Detail & Related papers  (2023-03-16T11:02:46Z)
• PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination [52.991211077362586]
  We propose a policy ensemble method to increase the diversity of partners in the population. We then develop a context-aware method enabling the ego agent to analyze and identify the partner's potential policy primitives. In this way, the ego agent is able to learn more universal cooperative behaviors for collaborating with diverse partners.
  arXiv  Detail & Related papers  (2023-01-16T12:14:58Z)
• Coordination with Humans via Strategy Matching [5.072077366588174]
  We present an algorithm for autonomously recognizing available task-completion strategies by observing human-human teams performing a collaborative task. By transforming team actions into low dimensional representations using hidden Markov models, we can identify strategies without prior knowledge. Robot policies are learned on each of the identified strategies to construct a Mixture-of-Experts model that adapts to the task strategies of unseen human partners.
  arXiv  Detail & Related papers  (2022-10-27T01:00:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.