HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models

• URL: http://arxiv.org/abs/2604.01881v1
• Date: Thu, 02 Apr 2026 10:40:15 GMT
• Title: HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
• Abstract summary: HieraVid is a hierarchical pruning framework that progressively and dynamically reduces visual redundancy.<n>We conduct extensive experiments on four widely used video understanding benchmarks to evaluate the effectiveness of HieraVid.<n>With only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
• Score: 43.7747080569325
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.

Related papers

• VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs [28.026438743789907]
  VideoScaffold is a dynamic representation framework designed for streaming video understanding.<n>It adaptively adjusts event granularity according to video duration while preserving fine-grained visual semantics.<n>The framework is modular and plug-and-play, seamlessly extending existing image-based MLLMs to continuous video comprehension.
  arXiv  Detail & Related papers  (2025-12-23T03:33:45Z)
• MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs [67.75865317787708]
  MMG-Vid is a training-free visual token pruning framework for video understanding.<n>We show that MMG-Vid can maintain over 99.5% of the original performance, while effectively reducing 75% visual tokens.
  arXiv  Detail & Related papers  (2025-08-28T17:50:03Z)
• Multi-Level LVLM Guidance for Untrimmed Video Action Recognition [0.0]
  This paper introduces the Event-Temporalized Video Transformer (ECVT), a novel architecture that bridges the gap between low-level visual features and high-level semantic information.<n>Experiments on ActivityNet v1.3 and THUMOS14 demonstrate that ECVT achieves state-of-the-art performance, with an average mAP of 40.5% on ActivityNet v1.3 and mAP@0.5 of 67.1% on THUMOS14.
  arXiv  Detail & Related papers  (2025-08-24T16:45:21Z)
• An Empirical Study on How Video-LLMs Answer Video Questions [41.97630658989303]
  Video Large Language Models (Video-LLMs) have shown strong capabilities in answering video questions.<n>To our knowledge, this is the first work to systematically uncover how Video-LLMs internally process and understand video content.
  arXiv  Detail & Related papers  (2025-08-21T08:42:35Z)
• Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos [53.723410664944566]
  We present Perceive Anything Model (PAM), a framework for comprehensive region-level visual understanding in images and videos.<n>Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation.<n>A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features into multi-modal tokens.
  arXiv  Detail & Related papers  (2025-06-05T17:51:39Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• The Devil is in Temporal Token: High Quality Video Reasoning Segmentation [68.33080352141653]
  Methods for Video Reasoning rely heavily on a single special token to represent the object in the video.<n>We propose VRS-HQ, an end-to-end video reasoning segmentation approach.<n>Our results highlight the strong temporal reasoning and segmentation capabilities of our method.
  arXiv  Detail & Related papers  (2025-01-15T03:17:24Z)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization [52.63845811751936]
  Video pre-training is challenging due to the modeling of its dynamics video. In this paper, we address such limitations in video pre-training with an efficient video decomposition. Our framework is both capable of comprehending and generating image and video content, as demonstrated by its performance across 13 multimodal benchmarks.
  arXiv  Detail & Related papers  (2024-02-05T16:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.