RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

• URL: http://arxiv.org/abs/2604.15736v1
• Date: Fri, 17 Apr 2026 06:22:41 GMT
• Title: RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees
• Abstract summary: We introduce RefereeBench, the first large-scale benchmark for evaluating MLLMs as automatic sports referees.<n>The benchmark is fully human-annotated to ensure high-quality annotations grounded in authentic officiating logic and multimodal evidence.<n>Our benchmark highlights the need for future MLLMs that better integrate domain knowledge and multimodal understanding.
• Score: 47.20504438153184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Multimodal Large Language Models (MLLMs) excel at generic video understanding, their ability to support specialized, rule-grounded decision-making remains insufficiently explored. In this paper, we introduce RefereeBench, the first large-scale benchmark for evaluating MLLMs as automatic sports referees. Spanning 11 sports with 925 curated videos and 6,475 QA pairs, RefereeBench evaluates five core officiating abilities: foul existence, foul and penalty classification, foul and penalty reasoning, entity perception, and temporal grounding. The benchmark is fully human-annotated to ensure high-quality annotations grounded in authentic officiating logic and multimodal evidence. Extensive evaluations of state-of-the-art MLLMs show that even the strongest models, such as Doubao-Seed-1.8 and Gemini-3-Pro, achieve only around 60% accuracy, while the strongest open-source model, Qwen3-VL, reaches only 47%. These results indicate that current models remain far from being reliable sports referees. Further analysis shows that while models can often identify incidents and involved entities, they struggle with rule application and temporal grounding, and frequently over-call fouls on normal clips. Our benchmark highlights the need for future MLLMs that better integrate domain knowledge and multimodal understanding, advancing trustworthy AI-assisted officiating and broader multimodal decision-making.

Related papers

• Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding [21.581174236498196]
  We introduce SportMV-Bench, a benchmark for multi-view sports video understanding.<n>Our analysis shows that current MLLMs fail to effectively exploit multi-view information.<n>We propose SportMV-Agent, an agentic framework that orchestrates an iterative loop of active view selection, perception tool execution, and evidence-grounded reasoning.
  arXiv  Detail & Related papers  (2026-07-13T17:34:36Z)
• SoccerRef-Agents: Multi-Agent System for Automated Soccer Refereeing [11.424559110766479]
  SoccerRef-Agents is a holistic and explainable multi-agent decision-making framework for soccer refereeing.<n>The main contributions are: (i) constructing the multimodal benchmark SoccerRefBench with over 1,200 referee theory questions and 600 foul video clips; (ii) building a vector-based knowledge base RefKnowledgeDB using the latest "Laws of the Game" and a classic case database for precise, knowledge-driven reasoning; and (iii) designing a novel multi-agent architecture that collaborates via cross-modal RAG to bridge the semantic gap between visual content and regulatory texts.
  arXiv  Detail & Related papers  (2026-04-25T17:46:59Z)
• Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation [34.28647703173823]
  Short-video platforms have become major channels for misinformation, where deceptive claims leverage visual experiments and social cues.<n>We introduce a comprehensive evaluation framework using a high-quality, manually annotated dataset of 200 short videos spanning four health domains.<n>This dataset provides fine-grained annotations for three deceptive patterns, experimental errors, logical fallacies, and fabricated claims.
  arXiv  Detail & Related papers  (2026-01-10T15:43:30Z)
• Self-Improving VLM Judges Without Human Annotations [74.29324865147838]
  We present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data.<n>Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on Multimodal RewardBench.<n>The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.
  arXiv  Detail & Related papers  (2025-12-02T20:52:19Z)
• MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs [61.70050081221131]
  MVU-Eval is the first comprehensive benchmark for evaluating Multi-Video Understanding for MLLMs.<n>Our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos.<n>These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics.
  arXiv  Detail & Related papers  (2025-11-10T16:02:33Z)
• MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI [59.196131618912005]
  Reasoning plays a crucial role in advancing Multimodal Large Language Models (MLLMs)<n>Existing MLLM benchmarks often fall short in precisely and comprehensively evaluating long-chain reasoning abilities.<n>We introduce MMReason, a new benchmark designed to precisely and comprehensively evaluate MLLM long-chain reasoning capability.
  arXiv  Detail & Related papers  (2025-06-30T07:14:38Z)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding [70.00000053847738]
  Real visual understanding is essential for AI systems that interact with the physical world.<n>Current evaluations mostly use real-world videos similar to training data.<n>We propose negative-control tests using videos that depict physically impossible or logically inconsistent events.
  arXiv  Detail & Related papers  (2025-05-02T15:58:38Z)
• Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models [82.92771279118888]
  We introduce Multimodal RewardBench, an expert-annotated benchmark for evaluating multimodal reward models.<n>Our dataset comprises 5,211 annotated (prompt, chosen response, rejected response) triplets collected from various vision-language models.<n>We find that even the top-performing models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall accuracy.
  arXiv  Detail & Related papers  (2025-02-20T01:48:13Z)
• Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Models [45.63440666848143]
  Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities.<n>Despite their success, MLLMs remain vulnerable to conversational adversarial inputs.<n>We study gaslighting negation attacks: a phenomenon where models, despite initially providing correct answers, are persuaded by user-provided negations to reverse their outputs.
  arXiv  Detail & Related papers  (2025-01-31T10:37:48Z)
• SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models [15.062299319625701]
  SPORTU is a benchmark designed to assess Multimodal Large Language Models (MLLMs) across multi-level sports reasoning tasks.<n>SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding.<n>SPORTU-video consists of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning.
  arXiv  Detail & Related papers  (2024-10-11T02:58:38Z)
• A Survey on Benchmarks of Multimodal Large Language Models [65.87641718350639]
  This paper presents a comprehensive review of 200 benchmarks and evaluations for Multimodal Large Language Models (MLLMs) We focus on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better.
  arXiv  Detail & Related papers  (2024-08-16T09:52:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.