HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming

• URL: http://arxiv.org/abs/2602.14214v1
• Date: Sun, 15 Feb 2026 16:13:51 GMT
• Title: HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming
• Abstract summary: HiVid is a framework to leverage Large Language Models to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming.<n>We show HiVid improves weight prediction accuracy by up to 11.5% for VOD and 26% for live streaming over SOTA baselines.
• Score: 21.627116007248777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm. (3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.

Related papers

• Harnessing Streaming Video in the Wild [53.23721420272668]
  Vision-Language Models (VLMs) are increasingly required to process video streams in applications such as video-call assistants, live commentary, and embodied robots.<n>An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing.<n>ExistingVLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment.
  arXiv  Detail & Related papers  (2026-06-07T13:00:19Z)
• VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models [39.78979236902648]
  Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames.<n>We propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants.
  arXiv  Detail & Related papers  (2026-04-08T22:31:20Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding [8.196520511153368]
  Video temporal grounding (VTG) is a critical task in video and understanding a key capability for extending large language models (Vid-LLMs) to broader applications.<n>Existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues.<n>We propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments.
  arXiv  Detail & Related papers  (2026-04-02T14:19:58Z)
• VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understanding [52.69880888587866]
  Current Video Large Language Models (Video LLMs) typically encode frames via a encoder vision and employ an autoregressive (AR) LLM for understanding and generation.<n>We propose VidLaDA, a Diffusion Video LLM based on Language Models (DLMs) that leverages bidirectional attention to unlock comprehensive modeling and decode tokens in parallel.<n>Experiments show VidLaDA rivals state-of-the-art AR baselines and outperforms DLM baselines, with MARS-Cache delivering over 12x speedup without compromising accuracy.
  arXiv  Detail & Related papers  (2026-01-25T15:02:01Z)
• TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs [81.78017865436816]
  We present TimeLens, a systematic investigation into building MLLMs with strong video temporal grounding ability.<n>We first expose critical quality issues in existing VTG benchmarks and introduce TimeLens-Bench, comprising meticulously re-annotated versions of three popular benchmarks.<n>We also address noisy training data through an automated re-annotation pipeline, yielding TimeLens-100K, a large-scale, high-quality training dataset.
  arXiv  Detail & Related papers  (2025-12-16T18:59:58Z)
• LiveStar: Live Streaming Assistant for Real-World Online Video Understanding [67.71551356747948]
  LiveStar is a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding.<n>LiveStar incorporates: (1) a training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) a response-silence decoding framework that determines optimal proactive response timing via a single forward pass verification; and (3) memory-aware acceleration via peak-end memory compression for online inference on 10+ minute videos, combined with streaming key-value cache to achieve 1.53x faster inference.
  arXiv  Detail & Related papers  (2025-11-07T15:00:37Z)
• UniVid: The Open-Source Unified Video Model [41.15980565061684]
  We present UniVid, a unified architecture that couples an MLLM with a diffusion decoder through a lightweight adapter.<n>Experiments on standard benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-29T02:31:36Z)
• Reasoning-Enhanced Domain-Adaptive Pretraining of Multimodal Large Language Models for Short Video Content Governance [34.134289344567705]
  We propose a reasoning-enhanced multimodal large language model (MLLM) pretraining paradigm for unified inappropriate content detection.<n>To address the distribution gap between short video content and the original pretraining data of MLLMs, we introduce three targeted pretraining tasks.<n> Experimental results show that our pretraining approach significantly improves the MLLM's performance in both zero-shot and supervised fine-tuning settings.
  arXiv  Detail & Related papers  (2025-09-25T19:46:34Z)
• Engagement Prediction of Short Videos with Large Multimodal Models [46.954597097369586]
  We empirically investigate the potential of large multimodal models (LMMs) for video engagement prediction.<n>VideoLLaMA2 processes key video frames, text-based metadata, and background sound, while Qwen2.5-VL utilizes only key video frames and text-based metadata.<n>By ensembling two types of models, our method achieves first place in the ICCV VQualA 2025 EVQA-SnapUGC Challenge on short-form video engagement prediction.
  arXiv  Detail & Related papers  (2025-08-04T15:21:29Z)
• PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling [95.2927277964409]
  We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams.<n>We propose the Predictive Future Modeling framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues.<n>Experiments show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters.
  arXiv  Detail & Related papers  (2025-05-29T06:46:19Z)
• SkyReels-V2: Infinite-length Film Generative Model [35.00453687783287]
  We propose SkyReels-V2, an Infinite-length Film Generative Model, that synergizes Multi-modal Large Language Model (MLLM), Multi-stage Pretraining, Reinforcement Learning, and Diffusion Forcing Framework.<n>We establish progressive-resolution pretraining for the fundamental video generation, followed by a four-stage post-training enhancement.
  arXiv  Detail & Related papers  (2025-04-17T16:37:27Z)
• Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding [57.26400319795876]
  Temporal Video Grounding (TVG) is a core challenge in long-form video understanding.<n>Recent Large Vision-Language Models (LVLMs) have shown early promise in tackling TVG through supervised fine-tuning.<n>We propose a novel post-training framework that enhances the generalization capabilities of LVLMs via reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-17T17:04:20Z)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding [70.7882058229772]
  This paper tackles an emerging and challenging problem of long video temporal grounding(VTG) Compared with short videos, long videos are also highly demanded but less explored. We propose CONE, an efficient COarse-to-fiNE alignment framework.
  arXiv  Detail & Related papers  (2022-09-22T10:58:42Z)
• Understanding Road Layout from Videos as a Whole [82.30800791500869]
  We formulate it as a top-view road attributes prediction problem and our goal is to predict these attributes for each frame both accurately and consistently. We exploit the following three novel aspects: leveraging camera motions in videos, including context cuesand incorporating long-term video information.
  arXiv  Detail & Related papers  (2020-07-02T00:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.