Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

• URL: http://arxiv.org/abs/2608.25529v1
• Date: Wed, 26 Aug 2026 08:35:42 GMT
• Title: Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
• Abstract summary: We introduce Video-IFBench, a benchmark for evaluating instruction following in video understanding.<n>We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions.<n>We build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples.
• Score: 61.926801508808616
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks focus primarily on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understanding, where models must satisfy diverse user-specified constraints, including those grounded in visual and audio content. We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions, covering 32 task types and 39 manually designed constraint categories spanning both semantic and format requirements. To reduce annotation cost, we build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples. We conduct a large-scale evaluation of more than 20 recent MLLMs and show that video instruction following remains challenging for current models, especially for instructions with many constraints, semantic constraints, or complex conditional structures that require selecting the correct branch or path based on video content. We hope our work will facilitate future research on instruction following in video understanding scenarios.

Related papers

• How Far Are Video Models from True Multimodal Reasoning? [32.10821745418955]
  CLVG-Bench is an evaluation framework designed to probe video models' zero-shot reasoning capabilities.<n>It comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories.<n>We propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations.
  arXiv  Detail & Related papers  (2026-04-21T08:04:02Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• IF-VidCap: Can Video Caption Models Follow Instructions? [44.2412700621584]
  We introduce IF-VidCap, a new benchmark for evaluating controllable video captioning.<n>IF-VidCap incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness.
  arXiv  Detail & Related papers  (2025-10-21T15:25:08Z)
• CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval [24.203328970223527]
  We present CaReBench, a testing benchmark for fine-grained video captioning and retrieval.<n>Uniquely, it provides manually separated spatial annotations and temporal annotations for each video.<n>Based on this design, we introduce two evaluation metrics, ReBias and CapST, specifically tailored for video retrieval and video captioning tasks.
  arXiv  Detail & Related papers  (2024-12-31T15:53:50Z)
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs [20.168429351519055]
  Video understanding is a crucial next step for multimodal large language models (LMLMs)<n>We propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.<n>We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities.
  arXiv  Detail & Related papers  (2024-06-13T17:50:05Z)
• Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning [102.54669633984278]
  We propose Momentor, a Video-LLM capable of accomplishing fine-grained temporal understanding tasks. We train Momentor on Moment-10M, enabling it to perform segment-level reasoning and localization.
  arXiv  Detail & Related papers  (2024-02-18T03:04:38Z)
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark [63.14000659130736]
  We introduce a comprehensive Multi-modal Video understanding Benchmark, namely MVBench. We first introduce a novel static-to-dynamic method to define these temporal-related tasks. Then, guided by the task definition, we automatically convert public video annotations into multiple-choice QA to evaluate each task.
  arXiv  Detail & Related papers  (2023-11-28T17:59:04Z)
• VidCoM: Fast Video Comprehension through Large Language Models with Multimodal Tools [44.78291853329394]
  textbfVidCoM is a fast adaptive framework that leverages Large Language Models (LLMs) to reason about videos using lightweight visual tools. An InsOVER algorithm locates the corresponding video events based on an efficient Hungarian matching between decompositions of linguistic instructions and video events.
  arXiv  Detail & Related papers  (2023-10-16T17:05:56Z)
• LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling [102.42424022921243]
  Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks. Experiments show that this unified framework achieves competitive performance on 14 VidL benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T20:43:25Z)
• Video Understanding as Machine Translation [53.59298393079866]
  We tackle a wide variety of downstream video understanding tasks by means of a single unified framework. We report performance gains over the state-of-the-art on several downstream tasks including video classification (EPIC-Kitchens), question answering (TVQA), captioning (TVC, YouCook2, and MSR-VTT)
  arXiv  Detail & Related papers  (2020-06-12T14:07:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.