Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs

• URL: http://arxiv.org/abs/2510.13251v1
• Date: Wed, 15 Oct 2025 07:59:06 GMT
• Title: Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
• Authors: Minji Kim, Taekyung Kim, Bohyung Han,
• Abstract summary: We investigate the internal information flow of VideoLLMs using mechanistic interpretability techniques.<n>Our analysis reveals consistent patterns across diverse VideoQA tasks.<n>These findings provide a blueprint on how VideoLLMs perform temporal reasoning.
• Score: 42.00309718904487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Large Language Models (VideoLLMs) extend the capabilities of vision-language models to spatiotemporal inputs, enabling tasks such as video question answering (VideoQA). Despite recent advances in VideoLLMs, their internal mechanisms on where and how they extract and propagate video and textual information remain less explored. In this study, we investigate the internal information flow of VideoLLMs using mechanistic interpretability techniques. Our analysis reveals consistent patterns across diverse VideoQA tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame interactions in early-to-middle layers, (2) followed by progressive video-language integration in middle layers. This is facilitated by alignment between video representations and linguistic embeddings containing temporal concepts. (3) Upon completion of this integration, the model is ready to generate correct answers in middle-to-late layers. (4) Based on our analysis, we show that VideoLLMs can retain their VideoQA performance by selecting these effective information pathways while suppressing a substantial amount of attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a blueprint on how VideoLLMs perform temporal reasoning and offer practical insights for improving model interpretability and downstream generalization. Our project page with the source code is available at https://map-the-flow.github.io

Related papers

• VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval [11.519642157641023]
  This paper focuses on leveraging MLLMs for video-text embedding and retrieval.<n>We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information.<n>We demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training.
  arXiv  Detail & Related papers  (2026-02-08T19:39:32Z)
• An Empirical Study on How Video-LLMs Answer Video Questions [41.97630658989303]
  Video Large Language Models (Video-LLMs) have shown strong capabilities in answering video questions.<n>To our knowledge, this is the first work to systematically uncover how Video-LLMs internally process and understand video content.
  arXiv  Detail & Related papers  (2025-08-21T08:42:35Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• SiLVR: A Simple Language-based Video Reasoning Framework [71.77141065418238]
  We present SiLVR, a Simple Language-based Video Reasoning framework.<n>In the first stage, SiLVR transforms raw video into language-based representations using multisensory inputs.<n>In the second stage, language descriptions are fed into a powerful reasoning LLM to solve complex video-language understanding tasks.
  arXiv  Detail & Related papers  (2025-05-30T17:59:19Z)
• Do Language Models Understand Time? [2.290956583394892]
  Large language models (LLMs) have revolutionized video-based computer vision applications, including action recognition, anomaly detection, and summarization.<n>This work critically examines the role of LLMs in video processing, with a specific focus on their temporal reasoning capabilities.<n>We analyze challenges posed by existing video datasets, including biases, lack of temporal annotations, and domain-specific limitations that constrain the temporal understanding of LLMs.
  arXiv  Detail & Related papers  (2024-12-18T13:38:06Z)
• ViLLa: Video Reasoning Segmentation with Large Language Model [48.75470418596875]
  We present ViLLa: Video reasoning segmentation with Large Language Model.<n>Our ViLLa manages to tackle these challenges through multiple core innovations.<n>To enable efficient processing of long videos, ViLLa incorporates (3) a key segment sampler that adaptively partitions long videos into shorter but semantically dense segments for less redundancy.
  arXiv  Detail & Related papers  (2024-07-18T17:59:17Z)
• ST-LLM: Large Language Models Are Effective Temporal Learners [58.79456373423189]
  Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation. How to effectively encode and understand videos in video-based dialogue systems remains to be solved. We propose ST-LLM, an effective video-LLM baseline with spatial-temporal sequence modeling inside LLM.
  arXiv  Detail & Related papers  (2024-03-30T10:11:26Z)
• VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
  We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
  arXiv  Detail & Related papers  (2023-12-04T19:48:02Z)
• Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA [96.10612095576333]
  We propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Our model is also comprised of dual-level attention (word/object and frame level), multi-head self-cross-integration for different sources (video and dense captions), and which pass more relevant information to gates. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-05-13T16:35:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.