SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions

• URL: http://arxiv.org/abs/2412.04036v1
• Date: Thu, 05 Dec 2024 10:19:36 GMT
• Title: SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions
• Authors: Bufang Yang, Yunqi Guo, Lilin Xu, Zhenyu Yan, Hongkai Chen, Guoliang Xing, Xiaofan Jiang,
• Abstract summary: SocialMind is the first proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas. We show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.
• Score: 3.7400236988012105
• License:
• Abstract: Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.

Related papers

• Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions [67.60397632819202]
  Building socially-intelligent AI agents (Social-AI) is a multidisciplinary, multimodal research goal. We identify a set of underlying technical challenges and open questions for researchers across computing communities to advance Social-AI.
  arXiv  Detail & Related papers  (2024-04-17T02:57:42Z)
• SocialBench: Sociality Evaluation of Role-Playing Conversational Agents [85.6641890712617]
  Large language models (LLMs) have advanced the development of various AI conversational agents. SocialBench is the first benchmark designed to evaluate the sociality of role-playing conversational agents at both individual and group levels. We find that agents excelling in individual level does not imply their proficiency in group level.
  arXiv  Detail & Related papers  (2024-03-20T15:38:36Z)
• SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents [73.35393511272791]
  We propose an interactive learning method, SOTOPIA-$pi$, improving the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement training on filtered social interaction data according to large language model (LLM) ratings.
  arXiv  Detail & Related papers  (2024-03-13T17:17:48Z)
• SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents [107.4138224020773]
  We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and humans. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We find that GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills.
  arXiv  Detail & Related papers  (2023-10-18T02:27:01Z)
• Social World Knowledge: Modeling and Applications [2.9417848476446364]
  Social world knowledge is a key ingredient in effective communication and information processing by humans and machines alike. We introduce SocialVec, a framework for eliciting low-dimensional entity embeddings from the social contexts in which they occur in social networks. Similar to word embeddings which facilitate tasks that involve text semantics, we expect the learned social entity embeddings to benefit multiple tasks of social flavor.
  arXiv  Detail & Related papers  (2023-06-28T15:25:30Z)
• Virtual Reality Training of Social Skills in Autism Spectrum Disorder: An Examination of Acceptability, Usability, User Experience, Social Skills, and Executive Functions [0.0]
  Poor social skills in autism spectrum disorder (ASD) are associated with reduced independence in daily life. Virtual reality (VR) may facilitate social skills training in social environments and situations to real life. Twenty-five participants with ASD attended a neuropsychological evaluation and three sessions of VR social skills training.
  arXiv  Detail & Related papers  (2023-04-15T07:54:37Z)
• Social Practices: a Complete Formalization [1.370633147306388]
  We present a formalization of a social framework for agents based on the concept of Social Practices. Social practices facilitate the practical reasoning of agents in standard social interactions. They also come with a social context that gives handles for social planning and deliberation.
  arXiv  Detail & Related papers  (2022-05-22T09:58:42Z)
• Social Behaviour Understanding using Deep Neural Networks: Development of Social Intelligence Systems [2.107969466194361]
  Social computing has evolved beyond social informatics toward the birth of social intelligence systems. This paper takes initiatives to propose a social behaviour understanding framework with the use of deep neural networks for social and behavioural analysis. Three systems, including depression detection, activity recognition and cognitive impairment screening, are developed to evidently demonstrate the importance of social intelligence.
  arXiv  Detail & Related papers  (2021-05-20T03:19:55Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)
• Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents [60.27066549589362]
  Social language used by human agents is associated with greater users' responsiveness and task completion. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
  arXiv  Detail & Related papers  (2020-12-29T08:22:48Z)
• Towards Social Identity in Socio-Cognitive Agents [0.0]
  We propose a socio-cognitive agent model based on the concept of Cognitive Social Frames. Cognitive Social Frames can be built around social groups, and form the basis for social group dynamics mechanisms and construct of Social Identity.
  arXiv  Detail & Related papers  (2020-01-20T15:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.