VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair

• URL: http://arxiv.org/abs/2607.14075v1
• Date: Wed, 15 Jul 2026 17:48:16 GMT
• Title: VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair
• Abstract summary: We propose an MLLM-based framework for visual software issue repair comprising two core modules: Image Type-aware Tool Calling (ITTC) and Dynamic Test-time Region Focusing (DTRF)<n>Experiments on the SWE-bench Multimodal benchmark demonstrate that VisualRepair consistently outperforms state-of-the-art approaches.
• Score: 39.86483414759863
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated Program Repair (APR) has witnessed significant progress with the advent of Large Language Models (LLMs). However, as modern software systems increasingly expose rich graphical user interfaces, effectively leveraging visual information from bug screenshots has become essential for understanding bugs and generating accurate fixes in multimodal scenarios. Real-world issue reports frequently contain heterogeneous visual attachments including UI screenshots, IDE snapshots, GIFs, and text-centric images, each with distinct visual patterns and domain-specific semantics that impose substantial perceptual demands on MLLMs. Furthermore, bug screenshots often contain large expanses of uninformative and bug-irrelevant regions, distracting the model's attention and limiting patch diversity. To address these challenges, we propose VisualRepair, an MLLM-based framework for visual software issue repair comprising two core modules: Image Type-aware Tool Calling (ITTC), which classifies input images and dynamically invokes a tailored tool-calling chain for robust visual interpretation, and Dynamic Test-time Region Focusing (DTRF), which grounds multiple bug-related region candidates and refines them via an adaptive zoom-in and zoom-out strategy to improve fault localization and promote diverse patch generation. Extensive experiments on the SWE-bench Multimodal benchmark demonstrate that VisualRepair consistently outperforms state-of-the-art approaches. VisualRepair resolves 196 and 25 instances on the test and dev sets, respectively, surpassing the best baseline by 10 and 11 instances. These results highlight the effectiveness of type-aware visual understanding and region-focused localization for automated visual software issue repair.

Related papers

• Learning to See What You Need: Gaze Attention for Multimodal Large Language Models [96.20985292033465]
  We introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation.<n>We show that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.
  arXiv  Detail & Related papers  (2026-05-13T06:54:09Z)
• GALA: Multimodal Graph Alignment for Bug Localization in Automated Program Repair [40.228155198574356]
  We propose GALA, a framework that shifts multimodal Automated Program Repair (APR) from implicit semantic guessing to explicit structural reasoning.<n>GALA operates in four stages: it first constructs an Image UI Graph to capture visual elements and their structural relationships; then performs file-level alignment by cross-referencing this UI graph with repository-level structures.<n>By enforcing both semantic and relational consistency across modalities, GALA establishes a highly accurate visual-to-code mapping.
  arXiv  Detail & Related papers  (2026-04-09T11:06:25Z)
• FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair [45.42550492783294]
  Multimodal Automated Program Repair (MAPR) extends traditional program repair by requiring models to jointly reason over source code, textual issue descriptions, and visual artifacts such as GUI screenshots.<n>We propose FailureMem, a multimodal repair framework that integrates three key mechanisms: a hybrid workflow-agent architecture that balances structured localization with flexible reasoning, active perception tools that enable region-level visual grounding, and a Failure Memory Bank that converts past repair attempts into reusable guidance.
  arXiv  Detail & Related papers  (2026-03-18T15:24:22Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• Chatting with Images for Introspective Visual Thinking [50.7747647794877]
  ''Chatting with images'' is a new framework that reframes visual manipulation as language-guided feature modulation.<n>Under the guidance of expressive language prompts, the model dynamically performs joint re-encoding over multiple image regions.<n>ViLaVT achieves strong and consistent improvements on complex multi-image and video-based spatial reasoning tasks.
  arXiv  Detail & Related papers  (2026-02-11T17:42:37Z)
• SVRepair: Structured Visual Reasoning for Automated Program Repair [17.545585659174773]
  Large language models (LLMs) have recently shown strong potential for Automated Program Repair (APR)<n>We propose textbfSVRepair, a multimodal APR framework with structured visual representation.<n> SVRepair first fine-tunes a vision-language model, textbfStructured Visual Representation (SVR), to uniformly transform heterogeneous visual artifacts into a emphsemantic scene graph.
  arXiv  Detail & Related papers  (2026-02-05T06:26:46Z)
• Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing [41.75392938686494]
  Large language model-(LLM) based automated program repair (APR) techniques have shown promising results in resolving real-world GitHub issue tasks.<n>These autonomous systems struggle to resolve multimodal problem scenarios due to limitations in interpreting and leveraging visual information.<n>We propose GUIRepair, a cross-modal reasoning approach for resolving multimodal issue scenarios by understanding and capturing visual information.
  arXiv  Detail & Related papers  (2025-06-19T08:42:11Z)
• Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining [67.87810796668981]
  Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL)<n>Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations.<n>These improvements translate to significant gains in both web and OS agent downstream tasks.
  arXiv  Detail & Related papers  (2024-12-13T18:40:10Z)
• More Pictures Say More: Visual Intersection Network for Open Set Object Detection [4.206612461069489]
  We introduce a strong DETR-based model, Visual Intersection Network for Open Set Object Detection (VINO) VINO constructs a multi-image visual bank to preserve the semantic intersections of each category across all time steps. Our approach guarantees a more precise alignment between target category semantics and region semantics, while significantly reducing pre-training time and resource demands.
  arXiv  Detail & Related papers  (2024-08-26T05:52:35Z)
• Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark [63.296342841358815]
  Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images.<n>The ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering.<n>We introduce MIRAGE, an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU.
  arXiv  Detail & Related papers  (2024-07-18T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.