Breaking the mould of Social Mixed Reality -- State-of-the-Art and Glossary

• URL: http://arxiv.org/abs/2507.23454v2
• Date: Fri, 01 Aug 2025 06:05:12 GMT
• Title: Breaking the mould of Social Mixed Reality -- State-of-the-Art and Glossary
• Authors: Marta Bieńkiewicz, Julia Ayache, Panayiotis Charalambous, Cristina Becchio, Marco Corragio, Bertram Taetz, Francesco De Lellis, Antonio Grotta, Anna Server, Daniel Rammer, Richard Kulpa, Franck Multon, Azucena Garcia-Palacios, Jessica Sutherland, Kathleen Bryson, Stéphane Donikian, Didier Stricker, Benoît Bardy,
• Abstract summary: This article explores a critical gap in Mixed Reality (MR) technology.<n>While advances have been made, MR still struggles to authentically replicate human embodiment and socio-motor interaction.<n>We advocate for MR systems that enhance social interaction and collaboration between humans and virtual autonomous agents.
• Score: 8.639663365908197
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This article explores a critical gap in Mixed Reality (MR) technology: while advances have been made, MR still struggles to authentically replicate human embodiment and socio-motor interaction. For MR to enable truly meaningful social experiences, it needs to incorporate multi-modal data streams and multi-agent interaction capabilities. To address this challenge, we present a comprehensive glossary covering key topics such as Virtual Characters and Autonomisation, Responsible AI, Ethics by Design, and the Scientific Challenges of Social MR within Neuroscience, Embodiment, and Technology. Our aim is to drive the transformative evolution of MR technologies that prioritize human-centric innovation, fostering richer digital connections. We advocate for MR systems that enhance social interaction and collaboration between humans and virtual autonomous agents, ensuring inclusivity, ethical design and psychological safety in the process.

Related papers

• Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interaction [4.276453870301421]
  We propose a Transformer-based multi-task learning framework to improve the decision-making process of social robots.<n>We construct a novel multi-party HRI dataset that captures real-world complexities, such as gaze misalignment.<n>Our findings contribute to the development of socially intelligent social robots capable of engaging in natural and context-aware multi-party interactions.
  arXiv  Detail & Related papers  (2025-07-15T03:42:14Z)
• Towards an LLM-powered Social Digital Twinning Platform [0.3499870393443268]
  Social Digital Twinner is a social simulation tool for exploring plausible effects of what-if scenarios in complex adaptive social systems.<n>The architecture is composed of three seamlessly integrated parts: a data infrastructure featuring real-world data and a synthetic population of citizens.<n>We demonstrate the tool's interactive capabilities by addressing the critical issue of youth school dropouts in Kragero, Norway.
  arXiv  Detail & Related papers  (2025-05-15T19:58:50Z)
• The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning [49.32390524168273]
  Our work aims to advance the social reasoning of embodied artificial intelligence (AI) agents in real-world social interactions.<n>We introduce a large-scale real-world Human Robot Social Interaction (HSRI) dataset to benchmark the capabilities of language models (LMs) and foundational models (FMs)<n>Our dataset consists of 400 real-world human social robot interaction videos and over 10K annotations, detailing the robot's social errors, competencies, rationale, and corrective actions.
  arXiv  Detail & Related papers  (2025-04-07T06:27:02Z)
• Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems [133.45145180645537]
  The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence.<n>As these agents increasingly drive AI research and practical applications, their design, evaluation, and continuous improvement present intricate, multifaceted challenges.<n>This survey provides a comprehensive overview, framing intelligent agents within a modular, brain-inspired architecture.
  arXiv  Detail & Related papers  (2025-03-31T18:00:29Z)
• Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation [70.52558242336988]
  We focus on predicting engagement in dyadic interactions by scrutinizing verbal and non-verbal cues, aiming to detect signs of disinterest or confusion. In this work, we collect a dataset featuring 34 participants engaged in casual dyadic conversations, each providing self-reported engagement ratings at the end of each conversation. We introduce a novel fusion strategy using Large Language Models (LLMs) to integrate multiple behavior modalities into a multimodal transcript''
  arXiv  Detail & Related papers  (2024-09-13T18:28:12Z)
• Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI [129.08019405056262]
  Embodied Artificial Intelligence (Embodied AI) is crucial for achieving Artificial Intelligence (AGI) MLMs andWMs have attracted significant attention due to their remarkable perception, interaction, and reasoning capabilities. In this survey, we give a comprehensive exploration of the latest advancements in Embodied AI.
  arXiv  Detail & Related papers  (2024-07-09T14:14:47Z)
• Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions [67.60397632819202]
  Building socially-intelligent AI agents (Social-AI) is a multidisciplinary, multimodal research goal. We identify a set of underlying technical challenges and open questions for researchers across computing communities to advance Social-AI.
  arXiv  Detail & Related papers  (2024-04-17T02:57:42Z)
• Extended Reality for Enhanced Human-Robot Collaboration: a Human-in-the-Loop Approach [2.336967926255341]
  Human-robot collaboration attempts to tackle these challenges by combining the strength and precision of machines with human ingenuity and perceptual understanding. We propose an implementation framework for an autonomous, machine learning-based manipulator that incorporates human-in-the-loop principles. The conceptual framework foresees human involvement directly in the robot learning process, resulting in higher adaptability and task generalization.
  arXiv  Detail & Related papers  (2024-03-21T17:50:22Z)
• On the Emergence of Symmetrical Reality [51.21203247240322]
  We introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. We propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality.
  arXiv  Detail & Related papers  (2024-01-26T16:09:39Z)
• CERN for AI: A Theoretical Framework for Autonomous Simulation-Based Artificial Intelligence Testing and Alignment [1.7586848727052424]
  This study investigates an innovative simulation-based multi-agent system within a virtual reality framework that replicates the real-world environment.<n>The framework is populated by automated 'digital citizens,' simulating complex social structures and interactions to examine and optimize AI.
  arXiv  Detail & Related papers  (2023-12-14T23:48:51Z)
• Towards socially-competent and culturally-adaptive artificial agents Expressive order, interactional disruptions and recovery strategies [0.0]
  The overarching aim of this work is to set a framework to make the artificial agent socially-competent beyond dyadic interaction-interaction. The paper highlights how this level of competence is achieved by focusing on just three dimensions: (i) social capability, (ii) relational role, and (iii) proximity.
  arXiv  Detail & Related papers  (2023-08-06T15:47:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.