Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning

• URL: http://arxiv.org/abs/2604.10895v1
• Date: Mon, 13 Apr 2026 01:56:00 GMT
• Title: Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning
• Abstract summary: We investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state)<n>Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically.<n>We propose a novel multi-task learning framework, termed as textbfSocialLDG that explicitly models the dynamic relationship among the states.
• Score: 7.393097938991832
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: For a robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as \textbf{SocialLDG} that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a language model to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-robot social interaction datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making.

Related papers

• Trajectory Learning with Graph Representations for Social Robot Navigation [1.0732935873226022]
  We propose an imitation learning framework that leverages social dynamics for socially compliant navigation.<n>We present a navigation module that captures temporal dynamics and mitigates error accumulations.<n>Our framework outperforms established data-driven baselines on simulation and a real-world dataset across diverse social metrics.
  arXiv  Detail & Related papers  (2026-06-21T23:57:54Z)
• The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanning [2.38295275136047]
  CRISP (Critique-and-Replan for Interactive Social Presence) is an autonomous framework where a robot critiques and replans its own actions.<n>CRISP integrates (1) extraction of movable joints and constraints by analyzing the robot's description file.<n>It can generate subtly different, human-like motions on various platforms using only the robot's structure file.
  arXiv  Detail & Related papers  (2026-03-20T17:40:21Z)
• Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction [16.018759023223502]
  We analyze how people signal interaction readiness through nonverbal behaviors and how expert wizards use these cues to guide engagement.<n>We propose a two-stage pipeline in which lightweight perceptual detectors are used to selectively trigger heavier video-based vision-language model (VLM) queries.<n>Our findings suggest that selectively using VLMs as proxies for social reasoning enables socially responsive robot behavior.
  arXiv  Detail & Related papers  (2025-12-08T05:17:58Z)
• Diffusion-Based Imitation Learning for Social Pose Generation [0.0]
  Intelligent agents, such as robots and virtual agents, must understand the dynamics of complex social interactions to interact with humans.<n>We explore how using a single modality, the pose behavior, of multiple individuals in a social interaction can be used to generate nonverbal social cues for the facilitator of that interaction.
  arXiv  Detail & Related papers  (2025-01-18T20:31:55Z)
• Visual-Geometric Collaborative Guidance for Affordance Learning [63.038406948791454]
  We propose a visual-geometric collaborative guided affordance learning network that incorporates visual and geometric cues. Our method outperforms the representative models regarding objective metrics and visual quality.
  arXiv  Detail & Related papers  (2024-10-15T07:35:51Z)
• Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task [17.190635800969456]
  In this paper, we examine using Large Language Models to infer human intention in a collaborative object categorization task with a physical robot.<n>We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions.
  arXiv  Detail & Related papers  (2024-04-12T12:15:14Z)
• Multi-Agent Dynamic Relational Reasoning for Social Robot Navigation [50.01551945190676]
  Social robot navigation can be helpful in various contexts of daily life but requires safe human-robot interactions and efficient trajectory planning. We propose a systematic relational reasoning approach with explicit inference of the underlying dynamically evolving relational structures. We demonstrate its effectiveness for multi-agent trajectory prediction and social robot navigation.
  arXiv  Detail & Related papers  (2024-01-22T18:58:22Z)
• Persistent-Transient Duality: A Multi-mechanism Approach for Modeling Human-Object Interaction [58.67761673662716]
  Humans are highly adaptable, swiftly switching between different modes to handle different tasks, situations and contexts. In Human-object interaction (HOI) activities, these modes can be attributed to two mechanisms: (1) the large-scale consistent plan for the whole activity and (2) the small-scale children interactive actions that start and end along the timeline. This work proposes to model two concurrent mechanisms that jointly control human motion.
  arXiv  Detail & Related papers  (2023-07-24T12:21:33Z)
• Data-driven emotional body language generation for social robotics [58.88028813371423]
  In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration. We implement a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions. The evaluation study found that the anthropomorphism and animacy of the generated expressions are not perceived differently from the hand-designed ones.
  arXiv  Detail & Related papers  (2022-05-02T09:21:39Z)
• Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition [54.749127627191655]
  The ability to recognize human partners is an important social skill to build personalized and long-term human-robot interactions. Deep learning networks have achieved state-of-the-art results and demonstrated to be suitable tools to address such a task. One solution is to make robots learn from their first-hand sensory data with self-supervision.
  arXiv  Detail & Related papers  (2021-03-16T13:50:24Z)
• Learning Asynchronous and Sparse Human-Object Interaction in Videos [56.73059840294019]
  Asynchronous-Sparse Interaction Graph Networks (ASSIGN) is able to automatically detect the structure of interaction events associated with entities in a video scene. ASSIGN is tested on human-object interaction recognition and shows superior performance in segmenting and labeling of human sub-activities and object affordances from raw videos.
  arXiv  Detail & Related papers  (2021-03-03T23:43:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.