VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models

• URL: http://arxiv.org/abs/2504.13122v1
• Date: Thu, 17 Apr 2025 17:39:41 GMT
• Title: VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models
• Authors: Haojian Huang, Haodong Chen, Shengqiong Wu, Meng Luo, Jinlan Fu, Xinya Du, Hanwang Zhang, Hao Fei,
• Abstract summary: We introduce VistaDPO, a framework for Video Hierarchical Spatial-Temporal Direct Preference Optimization.<n> VistaDPO enhances text-video preference alignment across three hierarchical levels.<n>Experiments on benchmarks such as Video Hallucination, Video QA, and Captioning performance tasks demonstrate that VistaDPO significantly improves the performance of existing LVMs.
• Score: 80.92928946973026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Video Models (LVMs) built upon Large Language Models (LLMs) have shown promise in video understanding but often suffer from misalignment with human intuition and video hallucination issues. To address these challenges, we introduce VistaDPO, a novel framework for Video Hierarchical Spatial-Temporal Direct Preference Optimization. VistaDPO enhances text-video preference alignment across three hierarchical levels: i) Instance Level, aligning overall video content with responses; ii) Temporal Level, aligning video temporal semantics with event descriptions; and iii) Perceptive Level, aligning spatial objects with language tokens. Given the lack of datasets for fine-grained video-language preference alignment, we construct VistaDPO-7k, a dataset of 7.2K QA pairs annotated with chosen and rejected responses, along with spatial-temporal grounding information such as timestamps, keyframes, and bounding boxes. Extensive experiments on benchmarks such as Video Hallucination, Video QA, and Captioning performance tasks demonstrate that VistaDPO significantly improves the performance of existing LVMs, effectively mitigating video-language misalignment and hallucination. The code and data are available at https://github.com/HaroldChen19/VistaDPO.

Related papers

• VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment [0.6854849895338531]
  Video-language models (Video-LLMs) excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. We introduce VideoPASTA, a framework that enhances Video-LLMs through targeted preference optimization.
  arXiv  Detail & Related papers  (2025-04-18T22:28:03Z)
• PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning [50.81779197183613]
  Direct Preference Optimization (DPO) helps reduce hallucinations in Video Multimodal Large Language Models (VLLMs) We propose Video Direct Preference Optimization (VDPO), an online preference learning framework that eliminates the need for preference annotation. We introduce Prompt-aware Multi-instance Learning VDPO, which selects augmentations based on prompt context.
  arXiv  Detail & Related papers  (2025-04-08T08:41:41Z)
• HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models [63.65066762436074]
  HiTVideo aims to address the potential limitations of existing video tokenizers in text-to-video generation tasks.<n>It utilizes a 3D causal VAE with a multi-layer discrete token framework, encoding video content into hierarchically structured codebooks.
  arXiv  Detail & Related papers  (2025-03-14T15:36:39Z)
• Temporal Preference Optimization for Long-Form Video Understanding [28.623353303256653]
  Temporal Preference Optimization (TPO) is a novel post-training framework designed to enhance the temporal grounding capabilities of video-LMMs.<n>TPO significantly enhances temporal understanding while reducing reliance on manually annotated data.<n>LLaVA-Video-TPO establishes itself as the leading 7B model on the Video-MME benchmark.
  arXiv  Detail & Related papers  (2025-01-23T18:58:03Z)
• VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos [58.765796160750504]
  VideoGLaMM is a new model for fine-grained pixel-level grounding in videos based on user-provided textual inputs.<n>The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions.<n> Experimental results show that our model consistently outperforms existing approaches across all three tasks.
  arXiv  Detail & Related papers  (2024-11-07T17:59:27Z)
• PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance [44.08446730529495]
  We propose a novel pooling strategy that simultaneously achieves token compression and instruction-aware visual feature aggregation. Our model is termed Prompt-guided Pooling LLaVA, or PPLLaVA for short.
  arXiv  Detail & Related papers  (2024-11-04T17:50:36Z)
• Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasets [62.280729345770936]
  We introduce the task of Alignable Video Retrieval (AVR) Given a query video, our approach can identify well-alignable videos from a large collection of clips and temporally synchronize them to the query. Our experiments on 3 datasets, including large-scale Kinetics700, demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-02T20:00:49Z)
• Training-free Video Temporal Grounding using Large-scale Pre-trained Models [41.71055776623368]
  Video temporal grounding aims to identify video segments within untrimmed videos that are most relevant to a given natural language query. Existing video temporal localization models rely on specific datasets for training and have high data collection costs. We propose a Training-Free Video Temporal Grounding approach that leverages the ability of pre-trained large models.
  arXiv  Detail & Related papers  (2024-08-29T02:25:12Z)
• VidLA: Video-Language Alignment at Scale [48.665918882615195]
  We propose VidLA, an approach for video-language alignment at scale. Our proposed approach surpasses state-of-the-art methods on multiple retrieval benchmarks.
  arXiv  Detail & Related papers  (2024-03-21T22:36:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.