Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization

• URL: http://arxiv.org/abs/2607.24354v1
• Date: Mon, 27 Jul 2026 12:31:58 GMT
• Title: Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
• Abstract summary: We introduce Cross-Modal Visual Feedback (CMVF)<n> CMVF incorporates (1) a failure visual diagnosis stage, in which a stronger VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns.<n>We show CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks.
• Score: 59.164665622252016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results. However, on multimodal tasks, the effectiveness of APO is fundamentally bottlenecked by a blind feedback channel: the optimizer reads the question, the prediction, and the gold answer, but never the input image on which the model failed, and therefore cannot diagnose visually grounded errors. As a remedy, we introduce Cross-Modal Visual Feedback (CMVF). CMVF incorporates (1) a failure-conditioned visual diagnosis stage, in which a stronger optimizer VLM inspects each failed image without access to predictions or labels, and (2) an error-aware aggregation stage that compresses these observations into reusable, task-level visual blind-spot patterns that drive the prompt rewrite. Crucially, the image is consumed only during optimization; the deployed artifact is an ordinary text prompt that runs at the same inference cost as any text-only baseline. Extensive results across 12 VQA datasets and 4 target VLMs demonstrate that CMVF consistently ranks first, improving over the strongest baseline on every target by 2.4 points on average, with gains of up to 6.5 points on individual benchmarks. Moreover, the optimizer self-organizes into expert-style visual checklists that transfer across models without re-optimization.

Related papers

• P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization [55.74731799669337]
  Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs)<n>Direct Preference Optimization (DPO) aims to learn directly from the corrected preferences provided by humans.<n>Existing preference pairs are often vision-agnostic and their inherently off-policy nature limits their effectiveness in guiding model learning.<n>We propose Perceptual Processing Direct Preference Optimization (P$2$-DPO), a novel training paradigm in which the model generates and learns from its own preference pairs.
  arXiv  Detail & Related papers  (2026-06-02T09:22:53Z)
• Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology [8.2867621159451]
  We propose GAUC, a training-free coreset selection method operating directly in the pre-trained multimodal embedding space.<n>On CRC-100K and MHIST across multiple open-source VLM architectures, GAUC consistently improves accuracy, calibration, and prompt gradient robustness.
  arXiv  Detail & Related papers  (2026-05-18T13:54:04Z)
• Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generation [47.96044455071274]
  V-QPP-Bench is the first benchmark dedicated to Visual Query Pre-processing.<n>Visual imperfections severely degrade both retrieval recall and end-to-end MRAG performance.<n>Off-the-shelf MLLMs struggle with tool selection and parameter prediction without specialized training.<n> supervised fine-tuning enables compact models to achieve comparable or superior performance to larger proprietary models.
  arXiv  Detail & Related papers  (2026-02-13T18:39:48Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization [50.13408999553116]
  We propose RL-Text2Vis, the first reinforcement learning framework for Text2Vis generation.<n>Our method uses a novel multi-objective reward that jointly optimize textual accuracy, code validity, and visualization quality.<n>Our results establish GRPO as an effective strategy for structured, multimodal reasoning in visualization generation.
  arXiv  Detail & Related papers  (2026-01-08T04:29:07Z)
• Input-Adaptive Visual Preprocessing for Efficient Fast Vision-Language Model Inference [2.8292841621378844]
  We propose an adaptive visual preprocessing method that adjusts input resolution and spatial coverage based on image content characteristics.<n>The proposed approach combines content-aware image analysis, adaptive resolution selection, and content-aware cropping to reduce visual redundancy prior to vision encoding.<n> Experimental results show that adaptive preprocessing reduces per-image inference time by over 50%, mean full generation time, and achieves a consistent reduction of more than 55% in visual token count.
  arXiv  Detail & Related papers  (2025-12-23T23:30:56Z)
• AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition [21.989922235835067]
  AdaptVision is an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach.<n>We show that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.
  arXiv  Detail & Related papers  (2025-12-03T13:43:30Z)
• Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models [43.35073848153914]
  Test-time prompt tuning for vision-language models has demonstrated impressive generalization capabilities under zero-shot settings.<n>We analyze the underlying causes of prompt optimization bias from both the model and data perspectives.
  arXiv  Detail & Related papers  (2025-11-12T09:35:31Z)
• Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement [102.22911097049953]
  Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks.<n>Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results.<n>We propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies.
  arXiv  Detail & Related papers  (2024-05-24T23:09:27Z)
• Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning [53.93074108238167]
  We construct Vision-Flan, the most diverse publicly available visual instruction tuning dataset to date. We propose a two-stage instruction tuning framework, in which VLMs are finetuned on Vision-Flan and further tuned on GPT-4 synthesized data. We find this two-stage tuning framework significantly outperforms the traditional single-stage visual instruction tuning framework.
  arXiv  Detail & Related papers  (2024-02-18T19:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.