A Multimodal Framework for Human-Multi-Agent Interaction

• URL: http://arxiv.org/abs/2603.23271v1
• Date: Tue, 24 Mar 2026 14:35:40 GMT
• Title: A Multimodal Framework for Human-Multi-Agent Interaction
• Abstract summary: We introduce a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent.<n>At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech.<n>Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
• Score: 5.410329948686681
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.

Related papers

• M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction [5.410329948686681]
  M2HRI is a multimodal multi-agent framework built on large language models that equips each robot with distinct personality and long-term memory.<n>Results show that both agent individuality and structured coordination are essential for coherent and socially appropriate multi-agent HRI.
  arXiv  Detail & Related papers  (2026-04-13T19:08:05Z)
• Advances and Innovations in the Multi-Agent Robotic System (MARS) Challenge [170.47383225329915]
  Multi-agent system frameworks are becoming essential for achieving scalable, efficient, and collaborative solutions.<n>This shift is fueled by three primary factors: increasing agent capabilities, enhancing system efficiency through task delegation, and enabling advanced human-agent interactions.<n>We propose the Multi-Agent Robotic System (MARS) Challenge, held at the NeurIPS 2025 Workshop on SpaVLE.
  arXiv  Detail & Related papers  (2026-01-26T17:56:19Z)
• Diffusion Forcing for Multi-Agent Interaction Sequence Modeling [52.769202433667125]
  MAGNet is a unified autoregressive diffusion framework for multi-agent motion generation.<n>It supports a wide range of interaction tasks through flexible conditioning and sampling.<n>It captures both tightly synchronized activities and loosely structured social interactions.
  arXiv  Detail & Related papers  (2025-12-19T18:59:02Z)
• InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs [72.5651722107621]
  InterAgent is an end-to-end framework for text-driven physics-based multi-agent humanoid control.<n>We introduce an autoregressive diffusion transformer equipped with multi-stream blocks, which decouples proprioception, exteroception, and action to cross-modal interference.<n>We also propose a novel interaction graph exteroception representation that explicitly captures fine-grained joint-to-joint spatial dependencies.
  arXiv  Detail & Related papers  (2025-12-08T10:46:01Z)
• Emergent Crowds Dynamics from Language-Driven Multi-Agent Interactions [6.149858590971363]
  We propose a novel method that exploits large language models (LLMs) to control agents' movement.<n>We validate our method in two complex scenarios that exemplify the interplay between social interactions, steering, and crowding.<n>Our framework produces more realistic crowd simulations, with emergent group behaviours arising naturally from any environmental setting.
  arXiv  Detail & Related papers  (2025-08-20T20:15:14Z)
• Agent AI: Surveying the Horizons of Multimodal Interaction [83.18367129924997]
  "Agent AI" is a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data. We envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.
  arXiv  Detail & Related papers  (2024-01-07T19:11:18Z)
• InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions [49.097973114627344]
  We present InterGen, an effective diffusion-based approach that incorporates human-to-human interactions into the motion diffusion process. We first contribute a multimodal dataset, named InterHuman. It consists of about 107M frames for diverse two-person interactions, with accurate skeletal motions and 23,337 natural language descriptions. We propose a novel representation for motion input in our interaction diffusion model, which explicitly formulates the global relations between the two performers in the world frame.
  arXiv  Detail & Related papers  (2023-04-12T08:12:29Z)
• Multi-robot Social-aware Cooperative Planning in Pedestrian Environments Using Multi-agent Reinforcement Learning [2.7716102039510564]
  We propose a novel multi-robot social-aware efficient cooperative planner that on the basis of off-policy multi-agent reinforcement learning (MARL) We adopt temporal-spatial graph (TSG)-based social encoder to better extract the importance of social relation between each robot and the pedestrians in its field of view (FOV)
  arXiv  Detail & Related papers  (2022-11-29T03:38:47Z)
• Rethinking Trajectory Prediction via "Team Game" [118.59480535826094]
  We present a novel formulation for multi-agent trajectory prediction, which explicitly introduces the concept of interactive group consensus. On two multi-agent settings, i.e. team sports and pedestrians, the proposed framework consistently achieves superior performance compared to existing methods.
  arXiv  Detail & Related papers  (2022-10-17T07:16:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.