TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos

• URL: http://arxiv.org/abs/2605.21443v1
• Date: Wed, 20 May 2026 17:32:26 GMT
• Title: TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos
• Abstract summary: We argue that some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames.<n>To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection.
• Score: 12.71084447285112
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames. A preliminary study confirms this gap, showing that temporal glitches are substantially harder for VLMs to detect than spatial ones. To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection. TempGlitch covers five temporal glitch types with balanced per-category samples, together with paired glitch-free videos that enable reliable binary evaluation. We evaluate 12 proprietary and open-weight VLMs across multiple frame-sampling settings. Our results show that current VLMs remain near chance on TempGlitch, often collapsing into either overly conservative behavior that misses most glitches or overly sensitive behavior that flags clean videos as glitchy. Moreover, denser frame sampling and larger model size do not reliably resolve these failures. TempGlitch provides a focused testbed for temporal reasoning, robust gameplay understanding, and automated glitch detection with VLMs. Code and data are available at the project website.

Related papers

• What's the Catch? Evaluating Temporal Consistency in Vision-Language Models [0.11121828451787184]
  Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal structure.<n>We introduce TimeCatch, where temporal anomalies are created by swapping consecutive frames and frame-level anomalies by replacing a frame with Gaussian noise.<n>Our evaluation reveals a substantial gap between frame-level and temporal anomaly detection.
  arXiv  Detail & Related papers  (2026-08-24T16:40:49Z)
• Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA [6.404338288187941]
  We study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance pipeline.<n>We benchmark six recent VLMs under a zero-shot prompting setting and analyse their sensitivity to four prompt variants.<n>Our results show that while the VLMs can capture visual cues associated with geometry clipping, they all produce substantial false positives on visually ambiguous frames.
  arXiv  Detail & Related papers  (2026-07-28T16:10:47Z)
• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events [52.031070006859544]
  Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.<n>We introduce Moment-Video, a benchmark for diagnosing the temporal fidelity of video MLLMs through momentary visual event understanding.
  arXiv  Detail & Related papers  (2026-06-01T17:32:20Z)
• Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding [86.37002814396674]
  We propose Foresee-to-Ground (F2G) for Video Temporal Grounding.<n>F2G integrates Predictive Temporal Perception with Evidence-Driven Reasoning.<n>It consistently improves grounding accuracy across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-05-21T04:03:25Z)
• RESP: Reference-guided Sequential Prompting for Visual Glitch Detection in Video Games [12.71084447285112]
  We present RESP, a practical multi-frame framework for gameplay glitch detection with vision-language models (VLMs)<n>For each test frame, we select a reference frame from earlier in the same video, establishing a visual baseline.<n> RESP aggregates noisy frame predictions into a stable video-level decision without fine-tuning the VLM.
  arXiv  Detail & Related papers  (2026-04-13T07:04:15Z)
• How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos [41.46982669196867]
  Video-based quality assurance (QA) for long-form gameplay is labor-intensive and error-prone.<n>Recent benchmarks suggest that vision language models (VLMs) can achieve promising results in detecting visual glitches on curated datasets.
  arXiv  Detail & Related papers  (2026-03-24T01:59:06Z)
• MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos [53.5354731425141]
  We introduce the task of early mistake detection in video.<n>The goal is to determine whether a keystep in a procedural activity is performed correctly while observing as little of the streaming video as possible.<n>We propose a method comprising a mistake detector and a reinforcement learning policy.
  arXiv  Detail & Related papers  (2026-03-15T07:14:49Z)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding [70.00000053847738]
  Real visual understanding is essential for AI systems that interact with the physical world.<n>Current evaluations mostly use real-world videos similar to training data.<n>We propose negative-control tests using videos that depict physically impossible or logically inconsistent events.
  arXiv  Detail & Related papers  (2025-05-02T15:58:38Z)
• MINERVA: Evaluating Complex Video Reasoning [72.12644008002566]
  We provide a new video reasoning dataset called MINERVA for modern multimodal models.<n>Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions.<n>We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors.
  arXiv  Detail & Related papers  (2025-05-01T17:41:49Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• GlitchBench: Can large multimodal models detect video game glitches? [6.817358575629576]
  We introduce GlitchBench, a novel benchmark derived from video game quality assurance tasks. Our benchmark is curated from a variety of unusual and glitched scenarios from video games. We show that GlitchBench presents a new challenge for state-of-the-art LMMs.
  arXiv  Detail & Related papers  (2023-12-08T18:14:21Z)
• Anomaly detection in surveillance videos using transformer based attention model [3.2968779106235586]
  This research suggests using a weakly supervised strategy to avoid annotating anomalous segments in training videos. The proposed framework is validated on real-world dataset i.e. ShanghaiTech Campus dataset.
  arXiv  Detail & Related papers  (2022-06-03T12:19:39Z)
• Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction [61.17654438176999]
  We propose a novel and robust unsupervised video anomaly detection method by frame prediction with proper design. Our proposed method obtains the frame-level AUROC score of 88.3% on the CUHK Avenue dataset.
  arXiv  Detail & Related papers  (2020-11-05T11:34:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.