MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

• URL: http://arxiv.org/abs/2606.07639v1
• Date: Mon, 01 Jun 2026 09:07:15 GMT
• Title: MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention
• Abstract summary: We show how a cross-attention backbone is better suited to real-time vision-language fusion than the prevailing decoder-only design.<n>Our model trails the strong Qwen2.5-VL-7B baseline overall.<n>On a single H200 with 256 frames per video, achieves it about a 5x speedup in time to first token.
• Score: 53.80990122465395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video understanding is shifting from the offline paradigm -- taking a fully recorded video as input and producing a single answer after it ends -- toward real-time interaction, in which the model perceives new frames while still replying, revises its answer as new evidence appears, and remains silent when there is nothing to say. We present MOSS-Video-Preview to validate this paradigm. Our central claim is that perception must not be blocked by generation; its natural realization is a two-channel architecture. We argue that a cross-attention backbone is better suited to real-time vision-language fusion than the prevailing decoder-only design: visual features enter through a side channel rather than joining the autoregressive sequence, so perception and generation run on separate, non-blocking pathways -- reducing the frequency of visual processing and exposing a clean channel-wise interface for independent compression. We complement this with a data synthesis pipeline that converts dense captions into real-time understanding QA whose answers are revised to match what the model has perceived so far, and we specialize an offline model on these data to elicit real-time behavior. Our model trails the strong Qwen2.5-VL-7B baseline overall -- a gap we attribute primarily to data and scale rather than the architecture -- yet attains competitive offline video and multimodal understanding, remains robust on the spatial and fine-grained temporal reasoning central to real-time use, and acquires behaviors that offline models lack: continuous perception, answer revision, and timely silence. On a single H200 with 256 frames per video, it achieves about a 5x speedup in time to first token and 2.7x higher decoding throughput, with negligible degradation in offline ability. Our study of paradigm, architecture, and data outlines a viable path toward real-time video understanding.

Related papers

• Compressing AI Traffic: Standardized Neural Network Coding of Visual-Token Representations in Split Vision-Language Inference [15.357950338341285]
  Video-MME accuracy remains close to the uncompressed reference up to a 98% reduction of the transmitted BF16 tensor and only then collapses.<n>Open-ended MLVU generation shows the same plateau-and- robustness profile under an LLM judge.
  arXiv  Detail & Related papers  (2026-09-01T13:07:32Z)
• Towards Error-Free Long Video Generation [56.86952045212838]
  In this paper, we introduce an infinite-length video generation framework that produces high-quality, dynamic, and identity-consistent single-shot long videos.<n>We first finetune a diffusion model as a video extension model on large-scale short video data to autoregressively generate temporally coherent clips.<n>Our framework establishes a new benchmark for realistic and coherent minute-level video synthesis.
  arXiv  Detail & Related papers  (2026-06-21T07:39:37Z)
• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
  We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
  arXiv  Detail & Related papers  (2026-06-05T07:29:20Z)
• WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs [37.61875409530676]
  We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order.<n>At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed.<n>These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints.
  arXiv  Detail & Related papers  (2026-02-25T17:45:45Z)
• DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression [38.495966630021556]
  We present DiffVC-RT, the first framework designed to achieve real-time diffusion-based Neural Video Compression (NVC)<n>We show that DiffVC-RT achieves 80.1% perceptual savings in terms of LPIPS over VTM-17.0 on HEVC dataset with real-time encoding and decoding speeds of 206 / 30 fps for 720p videos on an NVIDIA H800 GPU.
  arXiv  Detail & Related papers  (2026-01-28T12:59:25Z)
• TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding [14.570869250170139]
  TV-RAG is a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning.<n>By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning.
  arXiv  Detail & Related papers  (2025-12-29T14:10:22Z)
• DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation [61.59996525424585]
  DIFFVSGG is an online VSGG solution that frames this task as an iterative scene graph update problem.<n>We unify the decoding of object classification, bounding box regression, and graph generation three tasks using one shared feature embedding.<n>DIFFVSGG further facilitates continuous temporal reasoning, where predictions for subsequent frames leverage results of past frames as the conditional inputs of LDMs.
  arXiv  Detail & Related papers  (2025-03-18T06:49:51Z)
• VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers [23.541896057977745]
  VideoScan is an efficient vision-language model (VLM) inference framework for real-time video interaction.<n>VideoScan employs a single semantic carrier token to represent each frame.
  arXiv  Detail & Related papers  (2025-03-12T13:30:40Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• A Coding Framework and Benchmark towards Low-Bitrate Video Understanding [63.05385140193666]
  We propose a traditional-neural mixed coding framework that takes advantage of both traditional codecs and neural networks (NNs) The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved. We build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach.
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.