Bridging Vision Language Models and Symbolic Grounding for Video Question Answering

• URL: http://arxiv.org/abs/2509.11862v1
• Date: Mon, 15 Sep 2025 12:35:56 GMT
• Title: Bridging Vision Language Models and Symbolic Grounding for Video Question Answering
• Abstract summary: Video Question Answering (VQA) requires models to reason over spatial, temporal, and causal cues in videos.<n>Recent vision language models (VLMs) achieve strong results but often rely on shallow correlations, leading to weak temporal grounding and limited interpretability.<n>We study symbolic scene graphs (SGs) as intermediate grounding signals for VQA.<n>We introduce SG-VLM, a modular framework that integrates frozen VLMs with scene graph grounding via prompting and visual localization.
• Score: 4.215692222461999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Question Answering (VQA) requires models to reason over spatial, temporal, and causal cues in videos. Recent vision language models (VLMs) achieve strong results but often rely on shallow correlations, leading to weak temporal grounding and limited interpretability. We study symbolic scene graphs (SGs) as intermediate grounding signals for VQA. SGs provide structured object-relation representations that complement VLMs holistic reasoning. We introduce SG-VLM, a modular framework that integrates frozen VLMs with scene graph grounding via prompting and visual localization. Across three benchmarks (NExT-QA, iVQA, ActivityNet-QA) and multiple VLMs (QwenVL, InternVL), SG-VLM improves causal and temporal reasoning and outperforms prior baselines, though gains over strong VLMs are limited. These findings highlight both the promise and current limitations of symbolic grounding, and offer guidance for future hybrid VLM-symbolic approaches in video understanding.

Related papers

• Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models [4.644589778994777]
  Vision-language-action (VLA) models provide a unified paradigm for connecting visual perception, language understanding, and robotic control.<n>We propose a hierarchical long-horizon VLA architecture with an explicit language-memory module.<n>We evaluate the proposed method in multiple simulation environments and conduct sim-to-real experiments on a real robotic platform.
  arXiv  Detail & Related papers  (2026-08-05T12:32:15Z)
• VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization [34.39134324001484]
  We introduce a paradigm shift that transitions the role of Vision-Language Models (VLMs) to "teachers"<n>A VLM teacher extracts task-specific rules to differentiable rewards, formulate a V Reasoner via test-time online optimization of a LoRA module.<n>This strategy enables adaptive test-time optimization and extends the reasoning capabilities beyond the V's intrinsic boundaries.
  arXiv  Detail & Related papers  (2026-06-01T17:54:26Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models [43.09726338623949]
  Vision-Language-Action (VLA) models integrate pretrained large Vision-Language Models (VLM) into their policy backbone.<n>This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance.<n>We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters.
  arXiv  Detail & Related papers  (2026-01-06T09:58:24Z)
• VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation [18.15310805625469]
  VOST-SGG is a VLM-aided one-stage ST-SGG framework that integrates the common sense reasoning capabilities of vision-language models.<n>We propose a multi-modal feature bank that fuses visual, textual, and spatial cues for improved predicate classification.<n>Our approach achieves state-of-the-art performance, validating the effectiveness of integrating VLM-aided semantic priors and multi-modal features for ST-SGG.
  arXiv  Detail & Related papers  (2025-12-05T08:34:06Z)
• Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding [56.45689495743107]
  Vgent is a graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding.<n>We evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-15T19:14:58Z)
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning [51.15353027471834]
  iFinder is a semantic grounding framework that translates dash-cam videos into a hierarchical, interpretable data structure for large language models.<n>iFinder operates as a training-free pipeline that employs pretrained vision models to extract critical cues.<n>It significantly outperforms end-to-end V-VLMs on four zero-shot driving benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T20:25:53Z)
• SpotVLM: Cloud-edge Collaborative Real-time VLM based on Context Transfer [14.669949808424409]
  Vision-Language Models (VLMs) are increasingly deployed in real-time applications such as autonomous driving and human-computer interaction.<n>Existing systems commonly employ partitioned Large Vision-Language Models (LVLMs) or task offloading strategies.<n>We propose a novel cloud-edge collaborative paradigm forVLMs, termed Context Transfer, which treats the delayed outputs of LVLMs as historical context.
  arXiv  Detail & Related papers  (2025-08-18T05:51:41Z)
• Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts [65.04791072532106]
  We present LoCoVQA, a benchmark generator for evaluating long-context extractive reasoning in vision language models (VLMs) LoCoVQA augments test examples for mathematical reasoning, VQA, and character recognition tasks with increasingly long visual contexts. This test assesses how well VLMs can ignore irrelevant information when answering queries.
  arXiv  Detail & Related papers  (2024-06-24T17:58:03Z)
• RelationVLM: Making Large Vision-Language Models Understand Visual Relations [66.70252936043688]
  We present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations.
  arXiv  Detail & Related papers  (2024-03-19T15:01:19Z)
• Good Questions Help Zero-Shot Image Reasoning [110.1671684828904]
  Question-Driven Visual Exploration (QVix) is a novel prompting strategy that enhances the exploratory capabilities of large vision-language models (LVLMs) QVix enables a wider exploration of visual scenes, improving the LVLMs' reasoning accuracy and depth in tasks such as visual question answering and visual entailment. Our evaluations on various challenging zero-shot vision-language benchmarks, including ScienceQA and fine-grained visual classification, demonstrate that QVix significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2023-12-04T03:18:51Z)
• ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models [27.5219975853389]
  We find that pre-trained vision-and-language models (VLMs) and large language models (LLMs) are good at different kinds of visual commonsense reasoning problems. For problems where the goal is to infer conclusions beyond image content,VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well.
  arXiv  Detail & Related papers  (2023-10-09T17:10:35Z)
• Can I Trust Your Answer? Visually Grounded Video Question Answering [88.11169242115416]
  We study visually grounded VideoQA in response to the emerging trends of utilizing pretraining techniques for video-language understanding. We construct NExT-GQA -- an extension of NExT-QA with 10.5$K$ temporal grounding labels tied to the original QA pairs.
  arXiv  Detail & Related papers  (2023-09-04T03:06:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.