The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanning

• URL: http://arxiv.org/abs/2603.20164v1
• Date: Fri, 20 Mar 2026 17:40:21 GMT
• Title: The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanning
• Abstract summary: CRISP (Critique-and-Replan for Interactive Social Presence) is an autonomous framework where a robot critiques and replans its own actions.<n>CRISP integrates (1) extraction of movable joints and constraints by analyzing the robot's description file.<n>It can generate subtly different, human-like motions on various platforms using only the robot's structure file.
• Score: 2.38295275136047
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conventional robot social behavior generation has been limited in flexibility and autonomy, relying on predefined motions or human feedback. This study proposes CRISP (Critique-and-Replan for Interactive Social Presence), an autonomous framework where a robot critiques and replans its own actions by leveraging a Vision-Language Model (VLM) as a `human-like social critic.' CRISP integrates (1) extraction of movable joints and constraints by analyzing the robot's description file (e.g., MJCF), (2) generation of step-by-step behavior plans based on situational context, (3) generation of low-level joint control code by referencing visual information (joint range-of-motion visualizations), (4) VLM-based evaluation of social appropriateness and naturalness, including pinpointing erroneous steps, and (5) iterative refinement of behaviors through reward-based search. This approach is not tied to a specific robot API; it can generate subtly different, human-like motions on various platforms using only the robot's structure file. In a user study involving five different robot types and 20 scenarios, including mobile manipulators and humanoids, our proposed method achieved significantly higher preference and situational appropriateness ratings compared to previous methods. This research presents a general framework that minimizes human intervention while expanding the robot's autonomous interaction capabilities and cross-platform applicability. Detailed result videos and supplementary information regarding this work are available at: https://limjiyu99.github.io/inner-critic/

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• The Embodiment Gap in Robot Foundation Models [7.90590547085857]
  In robotics, a model can generalize while work still remains before it can run on a robot with a particular body.<n>We call the gap between reusable models, representations, or data and their use in execution on the target robot the embodiment gap.<n>This survey examines what can be reused across robot embodiments and what must still be implemented on a new robot.
  arXiv  Detail & Related papers  (2026-08-19T01:55:04Z)
• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• StARS: Socially Appropriate Robot Actions via a Recommender System-Driven Approach [9.290474217180407]
  We reformulate socially appropriate action generation as a preference modelling problem inspired by recommender systems.<n>We propose StARS, a novel model-agnostic framework that integrates collaborative filtering with learnable scene representations.<n>We evaluate StARS on two socially aware robotics datasets, MannersDB+ and SocNav1, and analyse robustness under sparse preference feedback.
  arXiv  Detail & Related papers  (2026-07-23T20:38:07Z)
• Detection and Recognition: A Pairwise Interaction Framework for Mobile Service Robots [2.895170564353903]
  Service robots need to reason about local human-human interactions to support safe and socially aware navigation.<n>We argue that pairwise human interaction constitute a minimal yet sufficient perceptual unit for robot-centric social understanding.<n>We adopt a two-stage framework in which candidate interacting pairs are first identified based on lightweight geometric and motion cues.
  arXiv  Detail & Related papers  (2026-02-25T19:12:07Z)
• From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection [57.74400052368147]
  This paper presents a social robot navigation framework that integrates geometric planning with contextual social reasoning.<n>The system first extracts obstacles and human dynamics to generate geometrically feasible candidate paths, then leverages a fine-tuned vision-language model (VLM) to evaluate these paths.<n>Experiments in four social navigation contexts demonstrate that our method achieves the best overall performance with the lowest personal space violation duration, the minimal pedestrian-facing time, and no social zone intrusions.
  arXiv  Detail & Related papers  (2026-02-09T18:46:12Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interaction [4.276453870301421]
  We propose a Transformer-based multi-task learning framework to improve the decision-making process of social robots.<n>We construct a novel multi-party HRI dataset that captures real-world complexities, such as gaze misalignment.<n>Our findings contribute to the development of socially intelligent social robots capable of engaging in natural and context-aware multi-party interactions.
  arXiv  Detail & Related papers  (2025-07-15T03:42:14Z)
• Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces [40.44502415484082]
  We propose using language to bridge the gap in human-like reasoning between perception and robot actions.<n>We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs)<n>We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset.
  arXiv  Detail & Related papers  (2024-12-30T23:59:30Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Robot Interaction Behavior Generation based on Social Motion Forecasting for Human-Robot Interaction [9.806227900768926]
  We propose to model social motion forecasting in a shared human-robot representation space. ECHO operates in the aforementioned shared space to predict the future motions of the agents encountered in social scenarios. We evaluate our model in multi-person and human-robot motion forecasting tasks and obtain state-of-the-art performance by a large margin.
  arXiv  Detail & Related papers  (2024-02-07T11:37:14Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Model Predictive Control for Fluid Human-to-Robot Handovers [50.72520769938633]
  Planning motions that take human comfort into account is not a part of the human-robot handover process. We propose to generate smooth motions via an efficient model-predictive control framework. We conduct human-to-robot handover experiments on a diverse set of objects with several users.
  arXiv  Detail & Related papers  (2022-03-31T23:08:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.