FuguReport
Daily Report / 2026-08-26

Daily Paper Briefing

2026-08-26 1 Paper

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

Authors Hongbo Liu, Peixian Chen, Sihan Liu, Peiyuan Zhang, Kai Zou, Dian Zheng, Xiaoxing Hu, Yuhao Dong, Mengdan Zhang, Yunhang Shen, Haoyu Cao, Wei Liu, Weibo Gu, Xing Sun, Shengjie Zhao
Affiliations Tencent / Tianjin University / Nanyang Technological University / Shanghai Jiao Tong University / The Chinese University of Hong Kong
Categories Evaluation / Instruction Following Evaluation / Multimodal LLMs on video tasks, Method / Data Construction / Semi-automatic pipeline with program and human verification, Application / Video Understanding / Multitask and nested instruction scenarios
License CC BY 4.0

Abstract Overview

Video-IFBench is a benchmark designed to evaluate how well multimodal large language models follow instructions in video understanding settings, rather than only measuring answer correctness. The benchmark uses four instruction templates—single-task, multi-task, selection, and nested instructions—to cover different levels of task composition and conditional complexity. It spans 32 task types and 39 response constraint categories, including both semantic and format constraints, and is built with a semi-automatic pipeline that combines MLLM-based extraction, programmatic processing, and human verification. The final benchmark contains 1.5K samples drawn from more than 700 public videos across diverse domains, and is evaluated with a hybrid protocol that combines LLM-as-Judge and deterministic verification.

Novelty

The paper's main novelty is introducing a dedicated benchmark for instruction following in video understanding, a capability that prior video benchmarks and multimodal instruction-following benchmarks have not systematically evaluated. It is also distinctive in combining complex conditional instruction structures with video-grounded semantic constraints and a hybrid evaluation pipeline.

Results

In evaluations on more than 20 recent MLLMs, the authors find that video instruction following remains difficult for current systems. The strongest overall model, Gemini-3-Pro, reaches 76.5 TCSR and 54.5 TISR, while the best open-source model, Qwen3.5-397B-A17B-Think, reaches 69.6 TCSR and 46.1 TISR. Performance drops notably for instructions with many constraints, semantic constraints, deeper nested structures, and more candidate branches in selection tasks.

Key Points

  1. Video-IFBench targets instruction adherence in video understanding using four templates: Single, Multi, Selection, and Nested.
  2. The benchmark contains 1.5K human-verified samples over more than 700 videos, with 32 task types and 39 constraint categories spanning semantic and format requirements.
  3. Empirical analysis shows that current MLLMs are especially weak on constraint-heavy and condition-dependent instructions, indicating that strong video understanding accuracy does not guarantee faithful instruction following.

References

Project
This page was created using generative AI such as GPT-5, Claude Opus 4, Gemini 3, Gemini 3.1 Flash Image, and their higher-end successor versions. No guarantee can be made regarding its contents.