STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation

• URL: http://arxiv.org/abs/2607.02922v1
• Date: Fri, 03 Jul 2026 03:28:25 GMT
• Title: STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation
• Abstract summary: Video segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries.<n>Existing methods address this through token compression, yet typically operate on features lacking temporal context.<n>State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $mathcalO(T)$ cost compression to reduce.
• Score: 60.597357847359255
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video reasoning segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries, producing dense spatiotemporal tokens whose quadratic self-attention cost makes long-video processing prohibitive. Existing methods address this through token compression, yet typically operate on encoder features lacking temporal context, constraining selection before content redundancy can be reliably assessed. Informed compression requires contextual awareness, but acquiring that awareness at full resolution incurs the same quadratic cost compression aims to reduce. State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $\mathcal{O}(T)$ cost, producing representations where content redundancy becomes assessable. Building on this, Selective SpatioTemporal Aggregation and Compression (STAC) enriches features via decoupled bidirectional spatial and causal temporal scanning, leveraging recurrence-derived redundancy for hierarchical compression with adaptive thresholds optimised with segmentation objective. STAC achieves 85% token reduction and 1.8$\times$ speedup while surpassing compression-free baselines on reasoning segmentation benchmarks in a zero-shot streaming-compatible setting. Code is available \href{https://github.com/MCG-NKU/nku-video}{here}.

Related papers

• Think in Sets for Streaming Video Token Compression [11.833562937219753]
  Streaming video makes causal, frame-by-frame pruning decisions, so modeling cross-frame interactions requires an explicit historical reference.<n>We introduce NovaCov, to our knowledge the first training-free, plug-and-play set-wise token compressor designed for streaming video.
  arXiv  Detail & Related papers  (2026-08-02T11:44:51Z)
• Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling [49.11379121512358]
  We propose a unified generative framework that leverages pre-trained Diffusion Transformer priors to achieve high perceptual quality at extremely lows.<n>Our method achieves state-of-the-art perceptual fidelity with rich spatial details and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-21T06:36:32Z)
• End-to-End Context Compression at Scale [81.70601323130997]
  Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.<n>Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.<n>Existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2026-06-08T15:43:16Z)
• Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting [18.71433885255431]
  Adaptive video tokenisation seeks to dynamically allocate token budgets based on the underlying visual complexity of a sequence.<n>We show that the latent space of a frozen continuous video tokeniser inherently encodes temporal redundancy that can be exploited directly.<n>We introduce a parameter-free adaptive token allocation mechanism that applies a fixed threshold to per-position temporal-L1 differences.
  arXiv  Detail & Related papers  (2026-06-04T13:31:12Z)
• OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models [39.853488828881986]
  OTT-Vid is a transport-derived allocation framework for temporal token compression.<n>We show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens.
  arXiv  Detail & Related papers  (2026-05-12T08:58:49Z)
• KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Models [12.489375650998554]
  KiToke is a training-free, query-agnostic token compression approach.<n>Our method estimates token diversity globally using a kernel-based measure.<n>Our experiments show that KiToke consistently outperforms existing training-free compression methods.
  arXiv  Detail & Related papers  (2026-04-03T19:18:22Z)
• Towards Practical Lossless Neural Compression for LiDAR Point Clouds [84.36825469211375]
  extreme sparsity of high-precision geometric details hinders efficient context modeling.<n>Our framework comprises two lightweight modules.<n>Experiments demonstrate competitive compression performance at real-time speed.
  arXiv  Detail & Related papers  (2026-03-26T10:02:07Z)
• Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention [23.015486635502437]
  Video language models (Video-LLMs) face high computational costs due to large volumes of visual tokens.<n>We propose a unified selection mechanism that weights semantic similarity to globally select tokens.<n>Unselected tokens are merged via clustering and refilled, preserving information integrity.<n>Our unifiedtemporal token compression strategy establishes the state-of-theart in video understanding under ultra-low token retention.
  arXiv  Detail & Related papers  (2026-03-23T13:15:22Z)
• Generative Video Compression with One-Dimensional Latent Representation [59.29493435530304]
  We introduce Generative Video Compression with One-Dimensional (1D) Latent Representation (GVC1D)<n>GVC1D encodes the video data into extreme compact 1D latent tokens conditioned on both short- and long-term contexts.<n> Experimental results indicate that GVC1D attains superior compression efficiency, surpassing previous video compression methods.
  arXiv  Detail & Related papers  (2026-03-16T14:02:17Z)
• Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generation [49.48204107529758]
  We define token overflow as a regime in which compressed representations no longer contain sufficient information to answer a given query.<n>In this paper, we find that query-agnostic saturation statistics reliably separate compressed from uncompressed token representations.<n>Lightweight probing classifiers over both query and context xRAG representations detect overflow with 0.72 AUC-ROC on average.<n>These results advance from query-independent diagnostics to query-aware detectors, enabling low-cost pre-LLM gating to mitigate compression-induced errors.
  arXiv  Detail & Related papers  (2026-02-12T18:15:08Z)
• VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction [55.66673587952058]
  Video understanding models are increasingly limited by the prohibitive storage and computational costs of large-scale datasets.<n>VideoCompressa is a novel framework for video data synthesis that reframes the problem as dynamic latent compression.
  arXiv  Detail & Related papers  (2025-11-24T07:07:58Z)
• DNN-Driven Compressive Offloading for Edge-Assisted Semantic Video Segmentation [7.9955920576321615]
  This paper introduces STAC, a uniform compression scheme tailored for edge-assisted semantic segmentation. We show that STAC can save up to 20.95% of bandwidth without losing accuracy, compared to the state-of-the-art algorithm.
  arXiv  Detail & Related papers  (2022-03-28T03:49:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.