Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration

• URL: http://arxiv.org/abs/2606.06388v2
• Date: Sat, 06 Jun 2026 00:32:51 GMT
• Title: Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration
• Abstract summary: We present ALMANAC, a dataset of action-level mental model ANnotations for Agent Collaboration.<n>ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations.<n>Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors.
• Score: 54.18136393879849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in LLM agents have enabled complex cognitive capabilities, such as multi-step reasoning, planning, and tool use, that increasingly position these agents as human collaborators. Effective collaboration, however, requires collaborators to continuously maintain and align mental models of their own reasoning,partners' intentions, and shared goals during the collaborative process. Today's agents rarely develop such capabilities since they are primarily optimized for task completion, and the community lacks authentic human collaboration data with action-level mental model annotations that could guide agents toward process-level collaborative competence. To bridge this gap, we present ALMANAC, a dataset of Action-Level Mental model ANnotations for Agent Collaboration built from the Map Task, a classic dyadic routing task from social science. ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations that record the participants' self-reasoning, perceived partner intent, and perceived team goal. We benchmark six LLMs on predicting humans' next-turn behavior and mental models. Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors and infer their underlying mental models.

Related papers

• CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views [82.15362247351763]
  We introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios.<n>The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects.<n>We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction.
  arXiv  Detail & Related papers  (2026-07-07T18:08:45Z)
• Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior [1.4016147623265656]
  Human-AI collaboration requires AI agents to understand human behavior for effective coordination.<n>This work examines whether embodied foundation model agents exhibit emergent collaborative behaviors indicating underlying mental models of their collaborators.
  arXiv  Detail & Related papers  (2026-05-05T15:20:15Z)
• CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks [55.84031315625504]
  CollabSkill is a framework for evaluating human-agent collaboration on real-world occupational tasks.<n>It pairs real human workers with AI agents on tasks matched to their occupational background.<n>Our analysis reveals that practical experience emerges as the primary driver of collaboration skill.
  arXiv  Detail & Related papers  (2026-04-20T03:03:25Z)
• Who is Helping Whom? Analyzing Inter-dependencies to Evaluate Cooperation in Human-AI Teaming [13.263258837438045]
  We propose the concept of constructive interdependence as a key metric for evaluating cooperation in human-agent teams.<n>Our results demonstrate that although trained agents attain high task rewards, they fail to induce cooperative behavior.<n>Our analysis reveals that teaming performance is not necessarily correlated with task reward, highlighting that task reward alone cannot reliably measure cooperation.
  arXiv  Detail & Related papers  (2025-02-10T19:16:20Z)
• Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration [50.657070334404835]
  Collaborative Gym is a framework enabling asynchronous, tripartite interaction among agents, humans, and task environments.<n>We instantiate Co-Gym with three representative tasks in both simulated and real-world conditions.<n>Our findings reveal that collaborative agents consistently outperform their fully autonomous counterparts in task performance.
  arXiv  Detail & Related papers  (2024-12-20T09:21:15Z)
• Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task [56.92961847155029]
  Theory of Mind (ToM) significantly impacts human collaboration and communication as a crucial capability to understand others. Mutual Theory of Mind (MToM) arises when AI agents with ToM capability collaborate with humans. We find that the agent's ToM capability does not significantly impact team performance but enhances human understanding of the agent.
  arXiv  Detail & Related papers  (2024-09-13T13:19:48Z)
• Large Language Model-based Human-Agent Collaboration for Complex Task Solving [94.3914058341565]
  We introduce the problem of Large Language Models (LLMs)-based human-agent collaboration for complex task-solving. We propose a Reinforcement Learning-based Human-Agent Collaboration method, ReHAC. This approach includes a policy model designed to determine the most opportune stages for human intervention within the task-solving process.
  arXiv  Detail & Related papers  (2024-02-20T11:03:36Z)
• MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming [27.911816995891726]
  We introduce MetaAgents, a social simulation framework populated with Large Language Models (LLMs)<n>We construct a job fair environment as a case study to scrutinize the team assembly and skill-matching behaviors of LLM-based agents.<n>Our evaluation demonstrates that LLM-based agents perform competently in making rational decisions to develop efficient teams.
  arXiv  Detail & Related papers  (2023-10-10T10:17:58Z)
• Investigating Agency of LLMs in Human-AI Collaboration Tasks [24.562034082480608]
  We build on social-cognitive theory to develop a framework of features through which Agency is expressed in dialogue. We collect a new dataset of 83 human-human collaborative interior design conversations.
  arXiv  Detail & Related papers  (2023-05-22T08:17:14Z)
• Behaviour-conditioned policies for cooperative reinforcement learning tasks [41.74498230885008]
  In various real-world tasks, an agent needs to cooperate with unknown partner agent types. Deep reinforcement learning models can be trained to deliver the required functionality but are known to suffer from sample inefficiency and slow learning. We suggest a method, where we synthetically produce populations of agents with different behavioural patterns together with ground truth data of their behaviour. We additionally suggest an agent architecture, which can efficiently use the generated data and gain the meta-learning capability.
  arXiv  Detail & Related papers  (2021-10-04T09:16:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.