Aggregating Visual Information with Optimal Transport for VideoLM Token Compression
- URL: http://arxiv.org/abs/2608.20473v2
- Date: Tue, 25 Aug 2026 09:52:24 GMT
- Title: Aggregating Visual Information with Optimal Transport for VideoLM Token Compression
- Abstract summary: Video language models process videos as dense visual-token sequences with substantial representational redundancy.<n>These sequences are therefore essential for reducing the visual-token burden on language-model decoding.<n>We introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations.
- Score: 55.867497983465476
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing the visual-token burden on language-model decoding. The central challenge is to preserve visual information dispersed across frames under such compression. To this end, we introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure. The resulting source-to-target coupling induces a distribution over source observations for each target support, directly specifying how the compressed video representation is constructed. We further adapt this construction along task and spatial axes. Question conditioning modulates the transport cost between source frames and target supports, while influencing how many supports are allocated to each temporal segment, thereby directing representation capacity toward question-relevant content. At multiple spatial granularities, AVIOT computes region-specific temporal transport plans and adaptively fuses the representations they yield, allowing different regions within the same compact representation to draw from different moments. Evaluations across varying compression ratios show that AVIOT matches or outperforms the uncompressed baseline on multiple video-understanding benchmarks while retaining strong performance at higher compression ratios.
Related papers
- Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding [49.540751364954666]
Visual Token Codec separates global and patch tokens into dedicated coding paths.<n>We show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks.
arXiv Detail & Related papers (2026-08-09T17:32:04Z) - OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models [39.853488828881986]
OTT-Vid is a transport-derived allocation framework for temporal token compression.<n>We show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens.
arXiv Detail & Related papers (2026-05-12T08:58:49Z) - FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding [55.700832127331324]
FLoC is an efficient visual token compression framework based on the facility location function.<n>Our method achieves remarkable efficiency gains by swiftly selecting a compact subset of tokens.<n>Our approach is training-free, model-agnostic, and query-agnostic, providing a versatile solution.
arXiv Detail & Related papers (2025-10-31T17:29:39Z) - State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding [50.866929044215965]
We propose a State Space Prompting (SSP) method for video understanding.<n>SSP combines intra-frame inter-frame prompts to aggregate and propagate keytemporal information in the video.<n>Our SSP significantly outperforms existing SOTA methods by 2.76% on average.
arXiv Detail & Related papers (2025-10-14T05:30:36Z) - Embedding Compression Distortion in Video Coding for Machines [67.97469042910855]
Currently, video transmission serves not only the Human Visual System (HVS) for viewing but also machine perception for analysis.<n>We propose a Compression Distortion Embedding (CDRE) framework, which extracts machine-perception-related distortion representation and embeds it into downstream models.<n>Our framework can effectively boost the rate-task performance of existing codecs with minimal overhead in terms of execution time, and number of parameters.
arXiv Detail & Related papers (2025-03-27T13:01:53Z) - PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models [64.9366388601049]
Visual token compression is leveraged to reduce the considerable token length of visual inputs.<n>We introduce a unified token compression strategy called Progressive Visual Token Compression.<n>Our model achieves state-of-the-art performance across various video understanding benchmarks.
arXiv Detail & Related papers (2024-12-12T18:59:40Z) - SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization [20.109136454526233]
We propose SweetTok, a novel video tokenizer to overcome the limitations in current video tokenization methods.<n>SweetTok compress visual inputs through distinct spatial and temporal queries via textbfDecoupled textbfAutotextbfEncoder (DQAE)<n>We show that SweetTok significantly improves video reconstruction results by textbf42.8% w.r.t rFVD on UCF-101 dataset.
arXiv Detail & Related papers (2024-12-11T13:48:06Z) - Neural-based Video Compression on Solar Dynamics Observatory Images [8.73521037463594]
NASA's Solar Dynamics Observatory (SDO) mission collects extensive data to monitor the Sun's daily activity.
Data compression plays a crucial role in addressing the challenges posed by limited telemetry rates.
This paper introduces a neural video compression technique that achieves a high compression ratio for the SDO's image data collection.
arXiv Detail & Related papers (2024-07-12T21:24:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.