M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

• URL: http://arxiv.org/abs/2604.11975v1
• Date: Mon, 13 Apr 2026 19:08:05 GMT
• Title: M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction
• Abstract summary: M2HRI is a multimodal multi-agent framework built on large language models that equips each robot with distinct personality and long-term memory.<n>Results show that both agent individuality and structured coordination are essential for coherent and socially appropriate multi-agent HRI.
• Score: 5.410329948686681
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot works treat robots as functionally identical, overlooking how robots individual identity shape user perception and how coordination shapes multi-robot behavior when such individuality is present. To address this, we introduce M2HRI, a multimodal multi-agent framework built on large language models that equips each robot with distinct personality and long-term memory, alongside a coordination mechanism conditioned on these differences. In a controlled user study (n = 105) in a multi-agent human-robot interaction (HRI) scenario, we find that LLM-driven personality traits are significantly distinguishable and enhance interaction quality, long-term memory improves personalization and preference awareness, and centralized coordination significantly reduces overlap while improving overall interaction quality. Together, these results demonstrate that both agent individuality and structured coordination are essential for coherent and socially appropriate multi-agent HRI. Project website and code are available at https://project-m2hri.github.io/.

Related papers

• CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy [67.76658561040625]
  CHORUS is a framework that adapts a single vision-language-action (VLA) backbone to control diverse, multi-robot teams.<n>In experiments, CHORUS achieves a 64% point improvement over decentralized, from-scratch models.<n>Results show that a shared VLA backbone is capable of achieving decentralized multi-robot collaboration.
  arXiv  Detail & Related papers  (2026-06-10T17:26:08Z)
• A Multimodal Framework for Human-Multi-Agent Interaction [5.410329948686681]
  We introduce a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent.<n>At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech.<n>Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
  arXiv  Detail & Related papers  (2026-03-24T14:35:40Z)
• Diffusion Forcing for Multi-Agent Interaction Sequence Modeling [52.769202433667125]
  MAGNet is a unified autoregressive diffusion framework for multi-agent motion generation.<n>It supports a wide range of interaction tasks through flexible conditioning and sampling.<n>It captures both tightly synchronized activities and loosely structured social interactions.
  arXiv  Detail & Related papers  (2025-12-19T18:59:02Z)
• Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interaction [4.276453870301421]
  We propose a Transformer-based multi-task learning framework to improve the decision-making process of social robots.<n>We construct a novel multi-party HRI dataset that captures real-world complexities, such as gaze misalignment.<n>Our findings contribute to the development of socially intelligent social robots capable of engaging in natural and context-aware multi-party interactions.
  arXiv  Detail & Related papers  (2025-07-15T03:42:14Z)
• EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents [33.77674812074215]
  We introduce a novel multi-agent framework designed to enable effective collaboration among heterogeneous robots.<n>We propose a self-prompted approach, where agents comprehend robot URDF files and call robot kinematics tools to generate descriptions of their physics capabilities.<n>The Habitat-MAS benchmark is designed to assess how a multi-agent framework handles tasks that require embodiment-aware reasoning.
  arXiv  Detail & Related papers  (2024-10-30T03:20:01Z)
• Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory [3.3906920519220054]
  We describe our approach to developing an intelligent and robust social robotic system for the Nadine platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models. This approach is novel compared to the current state-of-the-art LLM-based agents which do not implement human-like long-term memory or sophisticated emotional appraisal.
  arXiv  Detail & Related papers  (2024-05-30T15:55:41Z)
• Persistent-Transient Duality: A Multi-mechanism Approach for Modeling Human-Object Interaction [58.67761673662716]
  Humans are highly adaptable, swiftly switching between different modes to handle different tasks, situations and contexts. In Human-object interaction (HOI) activities, these modes can be attributed to two mechanisms: (1) the large-scale consistent plan for the whole activity and (2) the small-scale children interactive actions that start and end along the timeline. This work proposes to model two concurrent mechanisms that jointly control human motion.
  arXiv  Detail & Related papers  (2023-07-24T12:21:33Z)
• Proceeding of the 1st Workshop on Social Robots Personalisation At the crossroads between engineering and humanities (CONCATENATE) [37.838596863193565]
  This workshop aims to raise an interdisciplinary discussion on personalisation in robotics. It aims at bringing researchers from different fields together to propose guidelines for personalisation.
  arXiv  Detail & Related papers  (2023-07-10T11:11:24Z)
• Data-driven emotional body language generation for social robotics [58.88028813371423]
  In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration. We implement a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions. The evaluation study found that the anthropomorphism and animacy of the generated expressions are not perceived differently from the hand-designed ones.
  arXiv  Detail & Related papers  (2022-05-02T09:21:39Z)
• A ROS Architecture for Personalised HRI with a Bartender Social Robot [61.843727637976045]
  BRILLO project has the overall goal of creating an autonomous robotic bartender that can interact with customers while accomplishing its bartending tasks. We present the developed three-layers ROS architecture integrating a perception layer managing the processing of different social signals, a decision-making layer for handling multi-party interactions, and an execution layer controlling the behaviour of a complex robot composed of arms and a face.
  arXiv  Detail & Related papers  (2022-03-13T11:33:06Z)
• A MultiModal Social Robot Toward Personalized Emotion Interaction [1.2183405753834562]
  This study demonstrates a multimodal human-robot interaction (HRI) framework with reinforcement learning to enhance the robotic interaction policy. The goal is to apply this framework in social scenarios that can let the robots generate a more natural and engaging HRI framework.
  arXiv  Detail & Related papers  (2021-10-08T00:35:44Z)
• Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition [54.749127627191655]
  The ability to recognize human partners is an important social skill to build personalized and long-term human-robot interactions. Deep learning networks have achieved state-of-the-art results and demonstrated to be suitable tools to address such a task. One solution is to make robots learn from their first-hand sensory data with self-supervision.
  arXiv  Detail & Related papers  (2021-03-16T13:50:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.