HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models

• URL: http://arxiv.org/abs/2606.27187v1
• Date: Thu, 25 Jun 2026 15:50:33 GMT
• Title: HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models
• Abstract summary: Large vision-language models (LVLMs) have recently shown immense potential in automated content moderation.<n>HarmVideoBench is a diagnostic benchmark comprising 1,379 videos paired with 4,137 multiple-choice questions.<n>We introduce BCR, a benchmark-aligned method that predicts reasoning boundaries and dynamically retrieves context only when needed.
• Score: 28.84235879847985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (LVLMs) have recently shown immense potential in automated content moderation, sparking growing interest in developing harmful-video benchmarks. However, we identify two primary limitations in existing works: 1) The multi-layered characteristics of harmful videos are overlooked. Existing benchmarks predominantly formulate evaluation as a binary classification task, failing to capture implicit or deep contextual harms. 2) Explanatory rationales are completely absent. Current frameworks measure exclusively whether a model flags a video correctly rather than explaining why, turning evaluation into a black box where models can succeed through superficial shortcuts. To address these problems, we present HarmVideoBench, a multi-layered diagnostic benchmark comprising 1,379 videos paired with 4,137 multiple-choice questions. HarmVideoBench benchmarks three hierarchical dimensions: Observable Evidence, Clip-Internal Meaning, and Beyond-Clip Reasoning, aiming to evaluate models' deep understanding beyond surface cues with carefully balanced and curated samples. We evaluate 19 leading models on HarmVideoBench to assess their multidimensional understanding of harmful videos. Moreover, we introduce BCR, a benchmark-aligned method that predicts reasoning boundaries and dynamically retrieves context only when needed. Experimental results show that BCR substantially improves the base model's performance in harmful video understanding, raising the macro average from 61.7 percent to a state-of-the-art 84.4 percent.

Related papers

• How Far Are Video Models from True Multimodal Reasoning? [32.10821745418955]
  CLVG-Bench is an evaluation framework designed to probe video models' zero-shot reasoning capabilities.<n>It comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories.<n>We propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations.
  arXiv  Detail & Related papers  (2026-04-21T08:04:02Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models [93.73583158211115]
  Achieving fine-grained-temporal understanding in videos remains a major challenge for current Video Large Multimodels (Video LMMs)<n>We contribute in three core aspects: dataset, model, and benchmark.<n>First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically to enable joint learning of video understanding, grounding, and multi-turn video chat.<n>Second, we propose the SAMA model, which incorporates a versatile-temporal context aggregator and a Segment Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
  arXiv  Detail & Related papers  (2025-05-24T18:13:16Z)
• MINERVA: Evaluating Complex Video Reasoning [72.12644008002566]
  We provide a new video reasoning dataset called MINERVA for modern multimodal models.<n>Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions.<n>We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors.
  arXiv  Detail & Related papers  (2025-05-01T17:41:49Z)
• CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding [43.858197893052115]
  CG-Bench is a novel benchmark for clue-grounded question answering in long videos.<n>It features 1,219 manually curated videos categorized by a granular system with 14 primary categories, 171 secondary categories, and 638 tertiary categories.<n>The benchmark includes 12,129 QA pairs in three major question types: perception, reasoning, and hallucination.
  arXiv  Detail & Related papers  (2024-12-16T18:46:45Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Lost in Time: A New Temporal Benchmark for VideoLLMs [48.71203934876828]
  We show that the currently most used video-language benchmarks can be solved without requiring much temporal reasoning.<n>We propose TVBench, a novel open-source video multiple-choice question-answering benchmark.
  arXiv  Detail & Related papers  (2024-10-10T09:28:36Z)
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding [67.56182262082729]
  We introduce MMBench-Video, a quantitative benchmark to rigorously evaluate large vision-language models (LVLMs) in video understanding. MMBench-Video incorporates lengthy videos from YouTube and employs free-form questions, mirroring practical use cases. The benchmark is meticulously crafted to probe the models' temporal reasoning skills, with all questions human-annotated according to a carefully constructed ability taxonomy.
  arXiv  Detail & Related papers  (2024-06-20T17:26:01Z)
• VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs [64.60035916955837]
  VANE-Bench is a benchmark designed to assess the proficiency of Video-LMMs in detecting anomalies and inconsistencies in videos.<n>Our dataset comprises an array of videos synthetically generated using existing state-of-the-art text-to-video generation models.<n>We evaluate nine existing Video-LMMs, both open and closed sources, on this benchmarking task and find that most of the models encounter difficulties in effectively identifying the subtle anomalies.
  arXiv  Detail & Related papers  (2024-06-14T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.