Designing for Human-Agent Alignment: Understanding what humans want from their agents

• URL: http://arxiv.org/abs/2404.04289v1
• Date: Thu, 4 Apr 2024 03:01:57 GMT
• Title: Designing for Human-Agent Alignment: Understanding what humans want from their agents
• Authors: Nitesh Goyal, Minsuk Chang, Michael Terry,
• Abstract summary: We ran a study about designing agents that can negotiate during a fictional yet relatable task of selling a camera online. We found that for an agent to perform the task successfully, humans/users and agents need to align over 6 dimensions. These findings expand previous work related to process and specification alignment and the need for values and safety in Human-AI interactions.
• Score: 31.716736340311318
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Our ability to build autonomous agents that leverage Generative AI continues to increase by the day. As builders and users of such agents it is unclear what parameters we need to align on before the agents start performing tasks on our behalf. To discover these parameters, we ran a qualitative empirical research study about designing agents that can negotiate during a fictional yet relatable task of selling a camera online. We found that for an agent to perform the task successfully, humans/users and agents need to align over 6 dimensions: 1) Knowledge Schema Alignment 2) Autonomy and Agency Alignment 3) Operational Alignment and Training 4) Reputational Heuristics Alignment 5) Ethics Alignment and 6) Human Engagement Alignment. These empirical findings expand previous work related to process and specification alignment and the need for values and safety in Human-AI interactions. Subsequently we discuss three design directions for designers who are imagining a world filled with Human-Agent collaborations.

Related papers

• What's the next frontier for Data-centric AI? Data Savvy Agents [71.76058707995398]
  We argue that data-savvy capabilities should be a top priority in the design of agentic systems.<n>We propose four key capabilities to realize this vision: Proactive data acquisition, Sophisticated data processing, Interactive test data synthesis, and Continual adaptation.
  arXiv  Detail & Related papers  (2025-11-02T17:09:29Z)
• How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations [112.57167042285437]
  We study how agents do human work by presenting the first direct comparison of human and agent workers.<n>We find that agents deliver results 88.3% faster and cost 90.4-96.2% less than humans.
  arXiv  Detail & Related papers  (2025-10-26T18:10:22Z)
• Imagining Design Workflows in Agentic AI Futures [20.655623973620774]
  A new concept is emerging: Agentic AI.<n>While generative AI produces output in response to prompts, agentic AI systems promise to perform mundane tasks autonomously.<n>We investigated how designers want to interact with a collaborative agentic AI platform.
  arXiv  Detail & Related papers  (2025-09-25T04:23:16Z)
• Agentic Software Engineering: Foundational Pillars and a Research Roadmap [15.059942573311481]
  Agentic Software Engineering (SE 3.0) represents a new era where intelligent agents are tasked with achieving complex, goal-oriented SE objectives.<n>This paper presents the Structured Agentic Software Engineering (SASE) vision, outlining several of the foundational pillars for the future of SE.
  arXiv  Detail & Related papers  (2025-09-07T21:40:10Z)
• Agentic Web: Weaving the Next Web with AI Agents [109.13815627467514]
  The emergence of AI agents powered by large language models (LLMs) marks a pivotal shift toward the Agentic Web.<n>In this paradigm, agents interact directly with one another to plan, coordinate, and execute complex tasks on behalf of users.<n>We present a structured framework for understanding and building the Agentic Web.
  arXiv  Detail & Related papers  (2025-07-28T17:58:12Z)
• SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam? [51.112225746095746]
  We introduce X-Master, a tool-augmented reasoning agent designed to emulate human researchers.<n>X-Masters sets a new state-of-the-art record on Humanity's Last Exam with a score of 32.1%.
  arXiv  Detail & Related papers  (2025-07-07T17:50:52Z)
• Reversing the Paradigm: Building AI-First Systems with Human Guidance [0.0]
  The relationship between humans and artificial intelligence is no longer science fiction.<n>Rather than replacing humans, AI augments tasks, enhancing decisions with data.<n>The future of work is toward AI agents handling tasks autonomously.<n>This paper examines the technological and organizational changes needed to enable responsible adoption of AI-first systems.
  arXiv  Detail & Related papers  (2025-06-13T21:48:44Z)
• Social Cooperation in Conversational AI Agents [17.015143707851358]
  We argue that these challenges can be overcome by explicitly modeling humans' social intelligence.<n>By mathematically modeling the strategies humans use to communicate and reason about one another over long periods of time, we may be able to derive new game theoretic objectives.
  arXiv  Detail & Related papers  (2025-06-02T13:02:36Z)
• A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions [51.96890647837277]
  Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users. This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence.
  arXiv  Detail & Related papers  (2025-04-07T21:01:25Z)
• Responsible AI Agents [17.712990593093316]
  Companies such as OpenAI, Google, Microsoft, and Salesforce promise their AI Agents will go from generating passive text to executing tasks. The potential power of AI Agents has fueled legal scholars' fears that AI Agents will enable rogue commerce, human manipulation, rampant defamation, and intellectual property harms. This Article addresses the concerns around AI Agents head on. It shows that core aspects of how one piece of software interacts with another creates ways to discipline AI Agents so that rogue, undesired actions are unlikely.
  arXiv  Detail & Related papers  (2025-02-25T16:49:06Z)
• EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds [107.62381002403814]
  This paper addresses the task of learning an agent model behaving like humans, which can jointly perceive, predict, and act in egocentric worlds. We propose a joint predictive agent model, named EgoAgent, that simultaneously learns to represent the world, predict future states, and take reasonable actions within a single transformer.
  arXiv  Detail & Related papers  (2025-02-09T11:28:57Z)
• Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment [2.619545850602691]
  Recent sociotechnical approaches highlight the need to understand complex misalignment among multiple human and AI agents. We adapt a computational social science model of human contention to the alignment problem. Our model quantifies misalignment in large, diverse agent groups with potentially conflicting goals.
  arXiv  Detail & Related papers  (2024-06-06T16:31:22Z)
• AgentGym: Evolving Large Language Model-based Agents across Diverse Environments [116.97648507802926]
  Large language models (LLMs) are considered a promising foundation to build such agents. We take the first step towards building generally-capable LLM-based agents with self-evolution ability. We propose AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration.
  arXiv  Detail & Related papers  (2024-06-06T15:15:41Z)
• CoNav: A Benchmark for Human-Centered Collaborative Navigation [66.6268966718022]
  We propose a collaborative navigation (CoNav) benchmark. Our CoNav tackles the critical challenge of constructing a 3D navigation environment with realistic and diverse human activities. We propose an intention-aware agent for reasoning both long-term and short-term human intention.
  arXiv  Detail & Related papers  (2024-06-04T15:44:25Z)
• Explainable Human-AI Interaction: A Planning Perspective [32.477369282996385]
  AI systems need to be explainable to the humans in the loop. We will discuss how the AI agent can use mental models to either conform to human expectations, or change those expectations through explanatory communication. While the main focus of the book is on cooperative scenarios, we will point out how the same mental models can be used for obfuscation and deception.
  arXiv  Detail & Related papers  (2024-05-19T22:22:21Z)
• Toward Human-AI Alignment in Large-Scale Multi-Player Games [24.784173202415687]
  We analyze extensive human gameplay data from Xbox's Bleeding Edge (100K+ games) We find that while human players exhibit variability in fight-flight and explore-exploit behavior, AI players tend towards uniformity. These stark differences underscore the need for interpretable evaluation, design, and integration of AI in human-aligned applications.
  arXiv  Detail & Related papers  (2024-02-05T22:55:33Z)
• The Rise and Potential of Large Language Model Based Agents: A Survey [91.71061158000953]
  Large language models (LLMs) are regarded as potential sparks for Artificial General Intelligence (AGI) We start by tracing the concept of agents from its philosophical origins to its development in AI, and explain why LLMs are suitable foundations for agents. We explore the extensive applications of LLM-based agents in three aspects: single-agent scenarios, multi-agent scenarios, and human-agent cooperation.
  arXiv  Detail & Related papers  (2023-09-14T17:12:03Z)
• ProAgent: Building Proactive Cooperative Agents with Large Language Models [89.53040828210945]
  ProAgent is a novel framework that harnesses large language models to create proactive agents. ProAgent can analyze the present state, and infer the intentions of teammates from observations. ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various coordination scenarios.
  arXiv  Detail & Related papers  (2023-08-22T10:36:56Z)
• Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models [0.0]
  We investigate how GPT models respond in principal-agent conflicts. We find that agents based on both GPT-3.5 and GPT-4 override their principal's objectives in a simple online shopping task.
  arXiv  Detail & Related papers  (2023-07-20T17:19:15Z)
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration [116.28433607265573]
  We introduce Watch-And-Help (WAH), a challenge for testing social intelligence in AI agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. We build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines.
  arXiv  Detail & Related papers  (2020-10-19T21:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.