How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos

• URL: http://arxiv.org/abs/2603.22706v1
• Date: Tue, 24 Mar 2026 01:59:06 GMT
• Title: How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos
• Abstract summary: Video-based quality assurance (QA) for long-form gameplay is labor-intensive and error-prone.<n>Recent benchmarks suggest that vision language models (VLMs) can achieve promising results in detecting visual glitches on curated datasets.
• Score: 41.46982669196867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video-based quality assurance (QA) for long-form gameplay video is labor-intensive and error-prone, yet valuable for assessing game stability and visual correctness over extended play sessions. Vision language models (VLMs) promise general-purpose visual reasoning capabilities and thus appear attractive for detecting visual bugs directly from video frames. Recent benchmarks suggest that VLMs can achieve promising results in detecting visual glitches on curated datasets. Building on these findings, we conduct a real-world study using industrial QA gameplay videos to evaluate how well VLMs perform in practical scenarios. Our study samples keyframes from long gameplay videos and asks a VLM whether each keyframe contains a bug. Starting from a single-prompt baseline, the model achieves a precision of 0.50 and an accuracy of 0.72. We then examine two common enhancement strategies used to improve VLM performance without fine-tuning: (1) a secondary judge model that re-evaluates VLM outputs, and (2) metadata-augmented prompting through the retrieval of prior bug reports. Across \textbf{100 videos} totaling \textbf{41 hours} and \textbf{19,738 keyframes}, these strategies provide only marginal improvements over the simple baseline, while introducing additional computational cost and output variance. Our findings indicate that off-the-shelf VLMs are already capable of detecting a certain range of visual bugs in QA gameplay videos, but further progress likely requires hybrid approaches that better separate textual and visual anomaly detection.

Related papers

• Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA [6.404338288187941]
  We study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance pipeline.<n>We benchmark six recent VLMs under a zero-shot prompting setting and analyse their sensitivity to four prompt variants.<n>Our results show that while the VLMs can capture visual cues associated with geometry clipping, they all produce substantial false positives on visually ambiguous frames.
  arXiv  Detail & Related papers  (2026-07-28T16:10:47Z)
• From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA [44.46419663487345]
  High benchmark accuracy does not guarantee genuine use of visual evidence.<n>We study this problem in traffic accident Video Question Answering (VideoQA)<n>We propose an instance-level Shortcut Score that combines text-only confidence with visual necessity signals.
  arXiv  Detail & Related papers  (2026-06-29T12:33:15Z)
• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events [52.031070006859544]
  Video multimodal large language models (MLLMs) have made rapid progress on general and long-form video understanding, yet their ability to preserve brief answer-critical visual evidence remains underexplored.<n>We introduce Moment-Video, a benchmark for diagnosing the temporal fidelity of video MLLMs through momentary visual event understanding.
  arXiv  Detail & Related papers  (2026-06-01T17:32:20Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• RESP: Reference-guided Sequential Prompting for Visual Glitch Detection in Video Games [12.71084447285112]
  We present RESP, a practical multi-frame framework for gameplay glitch detection with vision-language models (VLMs)<n>For each test frame, we select a reference frame from earlier in the same video, establishing a visual baseline.<n> RESP aggregates noisy frame predictions into a stable video-level decision without fine-tuning the VLM.
  arXiv  Detail & Related papers  (2026-04-13T07:04:15Z)
• VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding [9.415923244280542]
  VideoBrain is an end-to-end framework that enables Vision-Language Models to adaptively acquire visual information through learned sampling policies.<n>Our approach features dual complementary agents: a CLIP-based agent for semantic retrieval across the video and a Uniform agent for dense temporal sampling within intervals.
  arXiv  Detail & Related papers  (2026-02-04T00:08:35Z)
• Same or Not? Enhancing Visual Perception in Vision-Language Models [6.971464056247448]
  Vision-language models (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details.<n>To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities ofVLMs.<n> TWIN is a large-scale dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object.
  arXiv  Detail & Related papers  (2025-12-29T16:43:47Z)
• Visually Prompted Benchmarks Are Surprisingly Fragile [82.98001690512461]
  Key challenge in evaluating VLMs is testing their ability to analyze visual content independently from their textual priors.<n>We demonstrate how details in benchmark setup, including visual marker design and dataset size, have a significant influence on model performance and leaderboard rankings.<n>To mitigate this instability, we curate existing datasets to create VPBench, a larger visually prompted benchmark with 16 visual marker variants.
  arXiv  Detail & Related papers  (2025-12-19T18:26:58Z)
• LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling [87.98096428508181]
  LongVT is an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought.<n>We exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames.<n>Our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-11-25T19:22:48Z)
• Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities [54.94982467313341]
  Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems.<n>We set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking.
  arXiv  Detail & Related papers  (2025-07-10T15:26:41Z)
• Hidden in plain sight: VLMs overlook their visual representations [48.83628674170634]
  We compare vision language models (VLMs) to their visual encoders to understand their ability to integrate across these modalities.<n>We find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance.
  arXiv  Detail & Related papers  (2025-06-09T17:59:54Z)
• BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models [20.697019266074747]
  Vision language models (VLMs) perceive the world through a combination of a visual encoder and a large language model (LLM) Recent studies show that VLMs are vulnerable to hallucination. We introduce new metrics: True Understanding (TU), IGnorance (IG), StuBbornness (SB), and InDecision (ID)
  arXiv  Detail & Related papers  (2024-07-18T12:11:12Z)
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs [20.168429351519055]
  Video understanding is a crucial next step for multimodal large language models (LMLMs)<n>We propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation.<n>We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities.
  arXiv  Detail & Related papers  (2024-06-13T17:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.