LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

• URL: http://arxiv.org/abs/2608.25729v1
• Date: Wed, 26 Aug 2026 12:43:43 GMT
• Title: LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding
• Abstract summary: LongVU-TTT inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM.<n>Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning.<n>LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
• Score: 30.902165004701683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.

Related papers

• DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding [11.584150931573738]
  Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries.<n>Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations.<n>We propose DSE-VTG, a framework that addresses both frame-wise visual encoding and fixed query embeddings.
  arXiv  Detail & Related papers  (2026-09-08T15:03:41Z)
• STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models [81.32710031596591]
  Video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames.<n>Existing methods built on vision-language models (LVLMs) often address this challenge by externalizing reasoning through chain-of-thought (CoT)<n>We propose STORMS, a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT.
  arXiv  Detail & Related papers  (2026-05-25T16:33:00Z)
• TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding [14.570869250170139]
  TV-RAG is a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning.<n>By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning.
  arXiv  Detail & Related papers  (2025-12-29T14:10:22Z)
• LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling [87.98096428508181]
  LongVT is an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought.<n>We exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames.<n>Our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-11-25T19:22:48Z)
• video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory [51.03819128505358]
  Video-SALMONN S is first to process 3-hour videos at 1 FPS and 360p resolution under a fixed memory budget.<n>A test-time-training memory module continually updates token representations to capture long-range dependencies.<n>A prompt-dependent memory reader retrieves context-relevant content from fixed-size memory.
  arXiv  Detail & Related papers  (2025-10-13T08:20:15Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding [55.320254859515714]
  ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench.<n>Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.
  arXiv  Detail & Related papers  (2024-12-29T15:42:24Z)
• FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding [25.21011724370177]
  Text-guided Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on textual descriptions.<n>We introduce FlashVTG, a framework featuring a Temporal Feature Layering (TFL) module and an Adaptive Score Refinement (ASR) module.<n>FlashVTG achieves state-of-the-art performance on four widely adopted datasets in both Moment Retrieval (MR) and Highlight Detection (HD)
  arXiv  Detail & Related papers  (2024-12-18T02:23:33Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.