Lightweight Visual Reasoning for Socially-Aware Robots

• URL: http://arxiv.org/abs/2603.03942v1
• Date: Wed, 04 Mar 2026 11:08:44 GMT
• Title: Lightweight Visual Reasoning for Socially-Aware Robots
• Abstract summary: We introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in Vision-Language Models (VLMs)<n>We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment, sequential scene description, and human-intention recognition.<n>Results show that our method improves Qwen 2.5 (7B) by $3.3%$ (less distance), $+0.057$ description score, and $+2.93%$ accuracy, with less than $3%$ extra parameters.
• Score: 41.776442767736604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by $3.3\%$ (less distance), $+0.057$ description score, and $+2.93\%$ accuracy, with less than $3\%$ extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains $+0.111,+0.055$ and $+10.81\%,+4.79\%$ on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics

Related papers

• WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation [49.31386176285509]
  WatchAct is a benchmark for robot manipulation grounded in observed human behavior.<n>WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains.<n>In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct.
  arXiv  Detail & Related papers  (2026-06-24T23:13:56Z)
• G$^3$VLA: Geometric inductive bias for Vision-Language-Action Models [3.704517635293094]
  Vision-language-action (VLA) models have made rapid progress in generalist robot manipulation.<n>Their visual tokens remain grounded in 2D image coordinates rather than the calibrated geometry of the robot's cameras.<n>We propose G$3$VLA, a camera-aware geometric module that injects calibrated structure into the visual-token stream of a pretrained VLA.
  arXiv  Detail & Related papers  (2026-06-23T12:02:36Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation [5.967530183571141]
  Human-to-Robot'' imitation learning pipeline enables robots to acquire manipulation skills directly from unstructured video demonstrations.<n>Key innovation is a modular framework that decouples the learning process into two distinct stages.<n>For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations.
  arXiv  Detail & Related papers  (2026-02-22T13:26:27Z)
• SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding [78.12178144115224]
  Robotic Foundation Models (RFMs) hold great promise as generalist, end-to-end systems for robot control.<n>We propose to enrich easy-to-collect non-robotic image data with 3D annotations and enhance a pretrained VLM with 3D understanding capabilities.<n>We introduce our main contribution, $textbfSPEAR-1$: a robotic foundation model that integrates grounded 3D perception with language-instructed embodied control.
  arXiv  Detail & Related papers  (2025-11-21T17:09:43Z)
• RoboOmni: Proactive Robot Manipulation in Omni-modal Context [165.09049429566238]
  We introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.<n>We present RoboOmni, a framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution.<n>Experiments in simulation and real-world settings show that Robo Omni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
  arXiv  Detail & Related papers  (2025-10-27T18:49:03Z)
• RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics [67.11221574129937]
  Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world.<n>We propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding.<n>RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-06-04T17:59:27Z)
• Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets [15.349107660229356]
  Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora.<n>We present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs.<n>We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories.
  arXiv  Detail & Related papers  (2025-05-21T13:42:52Z)
• Robotic Visual Instruction [10.110333805611974]
  We introduce the Robotic Visual Instruction (RoVI) to guide robotic tasks.<n>RoVI encodes spatial-temporal information into human-interpretable visual instructions.<n>We present Visual Instruction Embodied (VIEW) to enable robots to understand RoVI better.
  arXiv  Detail & Related papers  (2025-05-01T17:55:05Z)
• RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics [46.63773228934993]
  We introduce an automatic synthetic data generation pipeline that instruction-tunes vision language models (VLMs) to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks.
  arXiv  Detail & Related papers  (2024-06-15T19:22:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.