ViBR: Automated Bug Replay from Video-based Reports using Vision-Language Models

• URL: http://arxiv.org/abs/2604.19905v1
• Date: Tue, 21 Apr 2026 18:28:02 GMT
• Title: ViBR: Automated Bug Replay from Video-based Reports using Vision-Language Models
• Abstract summary: ViBR is a lightweight and fully automated approach that reproduces bugs directly from GUI recordings.<n>Results show that ViBR successfully reproduces 72% of bug recordings, significantly outperforming state-of-the-art baselines and ablation variants.
• Score: 17.094299668194378
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bug reports play a critical role in software maintenance by helping users convey encountered issues to developers. Recently, GUI screen capture videos have gained popularity as a bug reporting artifact due to their ease of use and ability to retain rich contextual information. However, automatically reproducing bugs from such recordings remains a significant challenge. Existing methods often rely on fragile image-processing heuristics, explicit touch indicators, or pre-constructed UI transition graphs, which require non-trivial instrumentation and app-specific setup. This paper presents ViBR, a lightweight and fully automated approach that reproduces bugs directly from GUI recordings. Specifically, ViBR combines CLIP-based embedding similarity for action boundary segmentation with Vision-Language Models (VLMs) for region-aware GUI state comparison and guided bug replay. Experimental results show that ViBR successfully reproduces 72% of bug recordings, significantly outperforming state-of-the-art baselines and ablation variants.

Related papers

• From Bug Reports to Browser-Executable Procedures: An LLM-Driven Agent for Web GUI Bug Reproduction [1.746500363342869]
  ReBug is a context-aware agent system that reconstructs, executes, and validates browser-level reproduction procedures from web GUI bug reports.<n>We evaluate ReBug on 667 real-world bug reports from four open-source web applications.
  arXiv  Detail & Related papers  (2026-08-04T12:50:43Z)
• A History-Aware Visually Grounded Critic for Computer Use Agents [70.50900668601969]
  HiViG is a test-time framework built around a multimodal critic trained on real GUI trajectories.<n>At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history.<n>Visually grounded critique verifies raw execution coordinates against the current screenshot to intercept errors before execution.
  arXiv  Detail & Related papers  (2026-06-09T16:39:10Z)
• SVRepair: Structured Visual Reasoning for Automated Program Repair [17.545585659174773]
  Large language models (LLMs) have recently shown strong potential for Automated Program Repair (APR)<n>We propose textbfSVRepair, a multimodal APR framework with structured visual representation.<n> SVRepair first fine-tunes a vision-language model, textbfStructured Visual Representation (SVR), to uniformly transform heterogeneous visual artifacts into a emphsemantic scene graph.
  arXiv  Detail & Related papers  (2026-02-05T06:26:46Z)
• BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos [63.03271511550633]
  BrokenVideos is a benchmark dataset of 3,254 AI-generated videos with meticulously annotated, pixel-level masks highlighting regions of visual corruption.<n>Our experiments show that training state of the art artifact detection models and multi modal large language models (MLLMs) on BrokenVideos significantly improves their ability to localize corrupted regions.
  arXiv  Detail & Related papers  (2025-06-25T03:30:04Z)
• BugRepro: Enhancing Android Bug Reproduction with Domain-Specific Knowledge Integration [4.833035081314386]
  BugRepro is a novel technique that integrates domain-specific knowledge to enhance the accuracy and efficiency of bug reproduction.<n>BugRepro significantly outperforms two state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-20T15:48:34Z)
• DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation [61.59996525424585]
  DIFFVSGG is an online VSGG solution that frames this task as an iterative scene graph update problem.<n>We unify the decoding of object classification, bounding box regression, and graph generation three tasks using one shared feature embedding.<n>DIFFVSGG further facilitates continuous temporal reasoning, where predictions for subsequent frames leverage results of past frames as the conditional inputs of LDMs.
  arXiv  Detail & Related papers  (2025-03-18T06:49:51Z)
• Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports [16.45808969240553]
  Video-based bug reports are increasingly being used to document bugs for programs centered around a graphical user interface (GUI) We introduce a new approach, called JANUS, that adapts the scene-learning capabilities of vision transformers to capture subtle visual and textual patterns that manifest on app UI screens. Janus also makes use of a video alignment technique capable of adaptive weighting of video frames to account for typical bug manifestation patterns.
  arXiv  Detail & Related papers  (2024-07-11T15:48:36Z)
• Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps [26.96558418166514]
  This paper proposes a novel vision-driven, multi-agent collaborative automated GUI testing approach for detecting non-crash functional bugs.<n>We evaluate Trident on 590 non-crash bugs and compare it with 12 baselines, it can achieve more than 14%-112% and 108%-147% boost in average recall and precision.
  arXiv  Detail & Related papers  (2024-07-03T11:58:09Z)
• VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text [80.24176572093512]
  We introduce Visual Caption Restoration (VCR), a vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images.<n>This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images.
  arXiv  Detail & Related papers  (2024-06-10T16:58:48Z)
• Using Developer Discussions to Guide Fixing Bugs in Software [51.00904399653609]
  We propose using bug report discussions, which are available before the task is performed and are also naturally occurring, avoiding the need for additional information from developers. We demonstrate that various forms of natural language context derived from such discussions can aid bug-fixing, even leading to improved performance over using commit messages corresponding to the oracle bug-fixing commits.
  arXiv  Detail & Related papers  (2022-11-11T16:37:33Z)
• BigIssue: A Realistic Bug Localization Benchmark [89.8240118116093]
  BigIssue is a benchmark for realistic bug localization. We provide a general benchmark with a diversity of real and synthetic Java bugs. We hope to advance the state of the art in bug localization, in turn improving APR performance and increasing its applicability to the modern development cycle.
  arXiv  Detail & Related papers  (2022-07-21T20:17:53Z)
• GLIB: Towards Automated Test Oracle for Graphically-Rich Applications [26.940850357831657]
  We propose textttGLIB based on a code-based data augmentation technique to detect game GUI glitches. textttGLIB can achieve 100% precision and 99.5% recall in detecting non-crashing bugs such as game GUI glitches.
  arXiv  Detail & Related papers  (2021-06-19T14:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.