IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models

• URL: http://arxiv.org/abs/2604.24002v1
• Date: Mon, 27 Apr 2026 03:34:33 GMT
• Title: IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models
• Abstract summary: This paper presents IntentVLM, a two-stage video-language framework for open-vocabulary human intention recognition.<n>IntrepidVLM is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation.<n>It achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance.
• Score: 5.381126267099941
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Improving the effectiveness of human-robot interaction requires social robots to accurately infer human goals through robust intention understanding. This challenge is particularly critical in multimodal settings, where agents must integrate heterogeneous signals including text, visual cues to form a coherent interpretation of user intent. This paper presents IntentVLM, a novel two-stage video-language framework designed for open-vocabulary human intention recognition. The approach is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation followed by structured inference through selection, effectively reducing hallucinations in latent reasoning. Evaluated on the IntentQA and Inst-IT Bench datasets, IntentVLM achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance. Our findings demonstrate that this structured reasoning approach enhances open-vocabulary intention understanding without catastrophic forgetting, offering a robust foundation for human-centered robotics.

Related papers

• GIVE: Grounding Human Gestures in Vision-Language-Action Models [38.010995423052975]
  GIVE (Gesture Intent via Visual-Semantic Enhancement) is an effective approach to enhance pre-trained Vision-Language-Action (VLA) models with human gesture understanding.<n>In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%.
  arXiv  Detail & Related papers  (2026-06-11T14:59:38Z)
• An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction [0.0]
  This work presents a novel HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic.<n>The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition.<n> Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75%.
  arXiv  Detail & Related papers  (2026-02-23T09:05:15Z)
• Stable Language Guidance for Vision-Language-Action Models [62.80963701282789]
  Residual Semantic Steering is a probabilistic framework that disentangles physical affordance from semantic execution.<n> RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.
  arXiv  Detail & Related papers  (2026-01-07T16:16:10Z)
• Explicit World Models for Reliable Human-Robot Collaboration [13.90528067304433]
  We take a radically different tack to the issue of reliable embodied AI.<n>We emphasise the dynamic, ambiguous and subjective nature of human-robot interactions.
  arXiv  Detail & Related papers  (2026-01-05T00:58:19Z)
• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs [85.69785384599827]
  Human-object interaction (HOI) detection aims to localize human-object pairs and the interactions between them.<n>Existing methods operate under a closed-world assumption, treating the task as a classification problem over a small, predefined verb set.<n>We propose GRASP-HO, a novel Generative Reasoning And Steerable Perception framework that reformulates HOI detection from the closed-set classification task to the open-vocabulary generation problem.
  arXiv  Detail & Related papers  (2025-12-19T14:41:50Z)
• IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interaction [51.130510883952546]
  Vision-Language-Action (VLA) models leverage pretrained vision-language models (VLMs) to couple perception with robotic control.<n>We propose textbfIntentionVLA, a VLA framework with a curriculum training paradigm and an efficient inference mechanism.<n>Our proposed method first leverages carefully designed reasoning data that combine intention inference, spatial grounding, and compact embodied reasoning.
  arXiv  Detail & Related papers  (2025-10-09T04:49:46Z)
• Open-Universe Assistance Games [6.21910767424247]
  We introduce GOOD, a data-efficient, online method that extracts goals in the form of natural language during an interaction with a human.<n> GOOD prompts an LLM to simulate users with different complex intents, using its responses to perform probabilistic inference over candidate goals.<n>We evaluate GOOD in a text-based grocery shopping domain and in a text-operated simulated household robotics environment.
  arXiv  Detail & Related papers  (2025-08-20T23:07:10Z)
• HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs [46.59239283399911]
  HumanSense is a benchmark designed to evaluate the human-centered perception and interaction capabilities of MLLMs.<n>Our evaluation reveals that leading MLLMs still have considerable room for improvement, particularly for advanced interaction-oriented tasks.<n>We employ a multi-stage, modality-progressive reinforcement learning to enhance the reasoning abilities of an Omni model.
  arXiv  Detail & Related papers  (2025-08-14T12:14:15Z)
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge [41.030494146004806]
  We propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling.<n>DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning.<n>Experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks.
  arXiv  Detail & Related papers  (2025-07-06T16:14:29Z)
• Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models [55.20626448358655]
  This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs) Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.
  arXiv  Detail & Related papers  (2023-11-07T08:27:32Z)
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
  INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
  arXiv  Detail & Related papers  (2021-08-25T07:35:21Z)
• You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
  The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
  arXiv  Detail & Related papers  (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.