TIME: Temporal-Sensitive Multi-Dimensional Instruction Tuning and Robust Benchmarking for Video-LLMs

• URL: http://arxiv.org/abs/2503.09994v2
• Date: Thu, 07 Aug 2025 14:15:07 GMT
• Title: TIME: Temporal-Sensitive Multi-Dimensional Instruction Tuning and Robust Benchmarking for Video-LLMs
• Authors: Yunxiao Wang, Meng Liu, Wenqi Liu, Xuemeng Song, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Guorui Zhou, Liqiang Nie,
• Abstract summary: Video large language models have achieved remarkable performance in tasks such as video question answering.<n>Our dataset focuses on enhancing temporal comprehension across five key dimensions.<n>We introduce a multi-task prompt fine-tuning approach that seamlessly integrates temporal-sensitive tasks into existing instruction datasets.
• Score: 56.92627816895305
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video large language models have achieved remarkable performance in tasks such as video question answering, however, their temporal understanding remains suboptimal. To address this limitation, we curate a dedicated instruction fine-tuning dataset that focuses on enhancing temporal comprehension across five key dimensions. In order to reduce reliance on costly temporal annotations, we introduce a multi-task prompt fine-tuning approach that seamlessly integrates temporal-sensitive tasks into existing instruction datasets without requiring additional annotations. Furthermore, we develop a novel benchmark for temporal-sensitive video understanding that not only fills the gaps in dimension coverage left by existing benchmarks but also rigorously filters out potential shortcuts, ensuring a more accurate evaluation. Extensive experimental results demonstrate that our approach significantly enhances the temporal understanding of video-LLMs while avoiding reliance on shortcuts.

Related papers

• SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM [36.28285195488772]
  Large language models (LLMs) have demonstrated exceptional capabilities in text understanding.<n>Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information.<n>This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding.
  arXiv  Detail & Related papers  (2026-02-03T14:39:16Z)
• Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding [56.45689495743107]
  Vgent is a graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding.<n>We evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-15T19:14:58Z)
• Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMs [54.502280390499756]
  We propose TimeWarp to create a targeted synthetic temporal dataset to fine-tune the model's responses to encourage it to focus on the given input video.<n>We demonstrate that when our method is applied to existing models, it significantly improves performance on temporal understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-04T21:48:40Z)
• Video-LLMs with Temporal Visual Screening [59.18455762289321]
  Temporal Visual Screening (TVS) is a new task that universally pre-processes video question answering and instruction tuning data.<n>TVS is formulated as a modular front-end adapter task that can be seamlessly integrated into both Video Instruction Tuning (training) and Video Question Answering (inference) pipelines.<n> Experiments demonstrate that incorporating TVS yields relative gains of 7.33% (training) and 34.6% (inference)
  arXiv  Detail & Related papers  (2025-08-27T14:33:32Z)
• DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs [5.074812070492738]
  We introduce DaMO, a data-efficient Video LLM specifically designed for accurate temporal reasoning and multimodal understanding.<n>We train DaMO via a structured four-stage progressive training paradigm, incrementally equipping the model with multimodal alignment, semantic grounding, and temporal reasoning capabilities.<n>Our work establishes a promising direction for data-efficient video-language modeling.
  arXiv  Detail & Related papers  (2025-06-13T08:13:05Z)
• Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency [56.475612147721264]
  We propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals.<n>We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA.<n>Results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs.
  arXiv  Detail & Related papers  (2025-06-02T17:28:26Z)
• DisTime: Distribution-based Time Representation for Video Large Language Models [23.176698643825123]
  DisTime is a lightweight framework designed to enhance temporal comprehension in Video-LLMs.<n>DisTime employs a learnable token to create a continuous temporal embedding space.<n>DisTime achieves state-of-the-art performance across benchmarks in three time-sensitive tasks.
  arXiv  Detail & Related papers  (2025-05-30T08:10:18Z)
• Moment Quantization for Video Temporal Grounding [29.081100914208974]
  We propose a novel Moment-Quantization based Video Temporal Grounding method (MQVTG) MQVTG quantizes the input video into various discrete vectors to enhance the discrimination between relevant and irrelevant moments. Our method effectively groups relevant features and separates irrelevant ones, aligning with our goal of enhancing discrimination.
  arXiv  Detail & Related papers  (2025-04-03T05:21:14Z)
• HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding [14.464718780172582]
  We introduce HierarQ, a task-aware hierarchical Q-Former based framework that sequentially processes frames to bypass the need for frame sampling.<n>We introduce a lightweight two-stream language-guided feature modulator to incorporate task awareness in video understanding.<n>Extensive evaluations on 10 video benchmarks across video understanding, question answering, and captioning tasks demonstrate HierarQ's state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-11T16:21:23Z)
• Token-Efficient Long Video Understanding for Multimodal LLMs [101.70681093383365]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Temporal Contrastive Learning for Video Temporal Reasoning in Large Vision-Language Models [44.99833362998488]
  Temporal Semantic Alignment via Dynamic Prompting (TSADP) is a novel framework that enhances temporal reasoning capabilities.<n>We evaluate TSADP on the VidSitu dataset, augmented with enriched temporal annotations.<n>Our analysis highlights the robustness, efficiency, and practical utility of TSADP, making it a step forward in the field of video-language understanding.
  arXiv  Detail & Related papers  (2024-12-16T02:37:58Z)
• On the Consistency of Video Large Language Models in Temporal Comprehension [57.985769348320616]
  Video large language models (Video-LLMs) can temporally ground language queries and retrieve video moments. We conduct a study on prediction consistency -- a key indicator for robustness and trustworthiness of temporal grounding.
  arXiv  Detail & Related papers  (2024-11-20T00:47:17Z)
• Temporal Reasoning Transfer from Text to Video [51.68487044397409]
  Video Large Language Models (Video LLMs) struggle with tracking temporal changes and reasoning about temporal relationships. We introduce the Textual Temporal reasoning Transfer (T3) to transfer temporal reasoning abilities from text to video domains. LongVA-7B model achieves competitive performance on comprehensive video benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T16:10:29Z)
• Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models [53.235170710385006]
  We introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge. In experiments, Grounded-VideoLLM excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA.
  arXiv  Detail & Related papers  (2024-10-04T10:04:37Z)
• HERMES: temporal-coHERent long-forM understanding with Episodes and Semantics [32.117677036812836]
  HERMES is a model that simulates episodic memory accumulation to capture action sequences. Episodic COmpressor efficiently aggregates crucial representations from micro to semi-macro levels. Semantic ReTRiever dramatically reduces feature dimensionality while preserving relevant macro-level information.
  arXiv  Detail & Related papers  (2024-08-30T17:52:55Z)
• Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval [0.17590081165362778]
  Video moment retrieval is a challenging task requiring fine-grained interactions between video and text modalities. Recent work in image-text pretraining has demonstrated that most existing pretrained models suffer from information asymmetry due to the difference in length between visual and textual sequences. We question whether the same problem also exists in the video-text domain with an auxiliary need to preserve both spatial and temporal information.
  arXiv  Detail & Related papers  (2023-12-12T17:00:46Z)
• Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models [58.17315970207874]
  We propose a zero-shot method for adapting generalisable visual-textual priors from arbitrary VLM to facilitate moment-text alignment. Experiments conducted on three VMR benchmark datasets demonstrate the notable performance advantages of our zero-shot algorithm.
  arXiv  Detail & Related papers  (2023-09-01T13:06:50Z)
• Distill and Collect for Semi-Supervised Temporal Action Segmentation [0.0]
  We propose an approach for the temporal action segmentation task that can simultaneously leverage knowledge from annotated and unannotated video sequences. Our approach uses multi-stream distillation that repeatedly refines and finally combines their frame predictions. Our model also predicts the action order, which is later used as a temporal constraint while estimating frames labels to counter the lack of supervision for unannotated videos.
  arXiv  Detail & Related papers  (2022-11-02T17:34:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.