Multi-Turn Human-LLM Interaction Through the Lens of a Two-Way Intelligibility Protocol

• URL: http://arxiv.org/abs/2410.20600v2
• Date: Fri, 12 Sep 2025 13:52:45 GMT
• Title: Multi-Turn Human-LLM Interaction Through the Lens of a Two-Way Intelligibility Protocol
• Authors: Harshvardhan Mestha, Karan Bania, Shreyas V, Sidong Liu, Ashwin Srinivasan,
• Abstract summary: We investigate a more structured approach based on an abstract protocol for interaction between agents.<n>The protocol is motivated by a notion of "two-way intelligibility" and is modelled by a pair of communicating finite-state machines.<n>The results provide evidence in support of the protocol's capability of capturing one- and two-way intelligibility in human-LLM interaction.
• Score: 1.5711521670164208
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Our interest is in the design of software systems involving a human-expert interacting -- using natural language -- with a large language model (LLM) on data analysis tasks. For complex problems, it is possible that LLMs can harness human expertise and creativity to find solutions that were otherwise elusive. On one level, this interaction takes place through multiple turns of prompts from the human and responses from the LLM. Here we investigate a more structured approach based on an abstract protocol described in [3] for interaction between agents. The protocol is motivated by a notion of "two-way intelligibility" and is modelled by a pair of communicating finite-state machines. We provide an implementation of the protocol, and provide empirical evidence of using the implementation to mediate interactions between an LLM and a human-agent in two areas of scientific interest (radiology and drug design). We conduct controlled experiments with a human proxy (a database), and uncontrolled experiments with human subjects. The results provide evidence in support of the protocol's capability of capturing one- and two-way intelligibility in human-LLM interaction; and for the utility of two-way intelligibility in the design of human-machine systems.

Related papers

• AgentMaster: A Multi-Agent Conversational Framework Using A2A and MCP Protocols for Multimodal Information Retrieval and Analysis [0.0]
  We present a pilot study of AgentMaster, a novel modular multi-protocol MAS framework with self-implemented A2A and MCP.<n>The system supports natural language interaction without prior technical expertise and responds to multimodal queries for tasks including information retrieval, question answering, and image analysis.<n>Overall, our proposed framework contributes to the potential capabilities of domain-specific, cooperative, and scalable conversational AI powered by MAS.
  arXiv  Detail & Related papers  (2025-07-08T03:34:26Z)
• Mapping Neural Signals to Agent Performance, A Step Towards Reinforcement Learning from Neural Feedback [2.9060647847644985]
  We introduce NEURO-LOOP, an implicit feedback framework that utilizes the intrinsic human reward system to drive human-agent interaction.<n>This work demonstrates the feasibility of a critical first step in the NEURO-LOOP framework: mapping brain signals to agent performance.<n>We conclude that a relationship between fNIRS data and agent performance exists using classical machine learning techniques.
  arXiv  Detail & Related papers  (2025-06-14T21:38:31Z)
• A Survey of AI Agent Protocols [35.431057321412354]
  There is no standard way for large language models (LLMs) agents to communicate with external tools or data sources. This lack of standardized protocols makes it difficult for agents to work together or scale effectively. A unified communication protocol for LLM agents could change this.
  arXiv  Detail & Related papers  (2025-04-23T14:07:26Z)
• Towards Anthropomorphic Conversational AI Part I: A Practical Framework [49.62013440962072]
  We introduce a multi- module framework designed to replicate the key aspects of human intelligence involved in conversations.<n>In the second stage of our approach, these conversational data, after filtering and labeling, can serve as training and testing data for reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-28T03:18:39Z)
• Building a Cognitive Twin Using a Distributed Cognitive System and an Evolution Strategy [0.5586073503694489]
  We show that it's possible to orchestrate many simple physical and virtual devices to achieve good approximations of a person's interaction behavior. The generated Cognitive Twin may later be used to automate tasks, generate more realistic human-like artificial agents or further investigate its behaviors.
  arXiv  Detail & Related papers  (2025-02-03T21:19:13Z)
• A Multimodal Automated Interpretability Agent [63.8551718480664]
  MAIA is a system that uses neural models to automate neural model understanding tasks. We first characterize MAIA's ability to describe (neuron-level) features in learned representations of images. We then show that MAIA can aid in two additional interpretability tasks: reducing sensitivity to spurious features, and automatically identifying inputs likely to be mis-classified.
  arXiv  Detail & Related papers  (2024-04-22T17:55:11Z)
• SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems [53.94772445896213]
  Large Language Model (LLM)-based multi-agent systems have demonstrated promising performance in simulating human society. We propose SpeechAgents, a multi-modal LLM based multi-agent system designed for simulating human communication.
  arXiv  Detail & Related papers  (2024-01-08T15:01:08Z)
• Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation [7.647395374489533]
  We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions. We show that our approach generates explanations as helpful as those produced by a human domain expert.
  arXiv  Detail & Related papers  (2023-11-29T20:16:23Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)
• Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models [55.20626448358655]
  This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs) Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.
  arXiv  Detail & Related papers  (2023-11-07T08:27:32Z)
• Explaining Agent Behavior with Large Language Models [7.128139268426959]
  We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions. We show how a compact representation of the agent's behavior can be learned and used to produce plausible explanations.
  arXiv  Detail & Related papers  (2023-09-19T06:13:24Z)
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods [86.80718559904854]
  This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. We evaluate methods that use pretrained language models to produce descriptions of function behavior in natural language and code.
  arXiv  Detail & Related papers  (2023-09-07T17:47:26Z)
• Exploring the Intersection of Large Language Models and Agent-Based Modeling via Prompt Engineering [0.0]
  Large language models (LLMs) have emerged as a potential solution to this bottleneck. We present two simulations of believable proxies of human behavior.
  arXiv  Detail & Related papers  (2023-08-14T18:58:00Z)
• Building Cooperative Embodied Agents Modularly with Large Language Models [104.57849816689559]
  We address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. We harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication.
  arXiv  Detail & Related papers  (2023-07-05T17:59:27Z)
• A Model for Intelligible Interaction Between Agents That Predict and Explain [1.335664823620186]
  We formalise the interaction model by taking agents to be automata with some special characteristics. We define One- and Two-Way Intelligibility as properties that emerge at run-time by execution of the protocol. We demonstrate using the formal model to: (a) identify instances of One- and Two-Way Intelligibility in literature reports on humans interacting with ML systems providing logic-based explanations, as is done in Inductive Logic Programming (ILP); and (b) map interactions between humans and machines in an elaborate natural-language based dialogue-model to One- or Two-Way Intellig
  arXiv  Detail & Related papers  (2023-01-04T20:48:22Z)
• DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations [119.1953397679783]
  We focus on advancing the state-of-the-art in interpreting multimodal models. Our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models.
  arXiv  Detail & Related papers  (2022-03-03T20:52:47Z)
• Multi-Agent Imitation Learning with Copulas [102.27052968901894]
  Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
  arXiv  Detail & Related papers  (2021-07-10T03:49:41Z)
• Challenging common interpretability assumptions in feature attribution explanations [0.0]
  We empirically evaluate the veracity of three common interpretability assumptions through a large scale human-subjects experiment. We find that feature attribution explanations provide marginal utility in our task for a human decision maker.
  arXiv  Detail & Related papers  (2020-12-04T17:57:26Z)
• Investigating Human Response, Behaviour, and Preference in Joint-Task Interaction [3.774610219328564]
  We have designed an experiment in order to examine human behaviour and response as they interact with Explainable Planning (XAIP) agents. We also present the results from an empirical analysis where we examined the behaviour of the two agents for simulated users.
  arXiv  Detail & Related papers  (2020-11-27T22:16:59Z)
• DRG: Dual Relation Graph for Human-Object Interaction Detection [65.50707710054141]
  We tackle the challenging problem of human-object interaction (HOI) detection. Existing methods either recognize the interaction of each human-object pair in isolation or perform joint inference based on complex appearance-based features. In this paper, we leverage an abstract spatial-semantic representation to describe each human-object pair and aggregate the contextual information of the scene via a dual relation graph.
  arXiv  Detail & Related papers  (2020-08-26T17:59:40Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)
• On the interaction between supervision and self-play in emergent communication [82.290338507106]
  We investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency. We find that first training agents via supervised learning on human data followed by self-play outperforms the converse.
  arXiv  Detail & Related papers  (2020-02-04T02:35:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.