Test-Time Hinting for Black-Box Vision-Language Models

• URL: http://arxiv.org/abs/2605.16410v1
• Date: Wed, 13 May 2026 14:35:22 GMT
• Title: Test-Time Hinting for Black-Box Vision-Language Models
• Abstract summary: Test-Time Hinting improves VLM performance via a single VLM call and requiring only black-box API access.<n>We show that Test-Time Hinting improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks.
• Score: 11.005886421208709
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time scaling (TTS) methods have proven highly effective for LLMs, yet their application to vision-language models (VLMs) remains relatively underexplored. Existing VLM TTS methods largely require open-weight model access or expensive repeated sampling, and are evaluated primarily on multimodal mathematical and scientific reasoning benchmarks rather than general visual understanding tasks. In this paper, we propose Test-Time Hinting, a method that improves VLM performance via a single VLM call and requiring only black-box API access, which makes it broadly applicable to frontier closed-weight models. Our method is motivated by the observation that VLM errors tend to cluster around recurring failure patterns. We therefore train a lightweight hint generator model to predict, for a given test input, which "hint" should be prepended to the prompt, providing targeted contextual or procedural guidance that steers the VLM away from its characteristic failure modes. We show that Test-Time Hinting improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks and that these gains generalize to unseen benchmarks and VLMs without retraining the hint generator.

Related papers

• DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection [39.153744982595036]
  We propose Detection Prompt Optimization (DetPO), a gradient-free test-time optimization approach.<n>Our proposed approach yields consistent improvements across generalist MLLMs on Roboflow20-VL and LVIS.
  arXiv  Detail & Related papers  (2026-03-24T17:26:55Z)
• TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs [81.78017865436816]
  We present TimeLens, a systematic investigation into building MLLMs with strong video temporal grounding ability.<n>We first expose critical quality issues in existing VTG benchmarks and introduce TimeLens-Bench, comprising meticulously re-annotated versions of three popular benchmarks.<n>We also address noisy training data through an automated re-annotation pipeline, yielding TimeLens-100K, a large-scale, high-quality training dataset.
  arXiv  Detail & Related papers  (2025-12-16T18:59:58Z)
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation [73.76465227729818]
  Open-source Vision-Language Models (VLMs) have achieved state-of-the-art performance on benchmark tasks.<n>Pretraining corpora raise a critical concern for both practitioners and users: inflated performance due to test-set leakage.<n>We show that existing detection approaches either fail outright or exhibit inconsistent behavior.<n>We propose a novel simple yet effective detection method based on multi-modal semantic perturbation.
  arXiv  Detail & Related papers  (2025-11-05T18:59:52Z)
• Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned [29.44294456857936]
  Process Reward Models (PRMs) improve the reliability of reasoning in large language models.<n>Existing Vision-Language PRMs rely on Monte Carlo Tree Search (MCTS) for data construction.<n>We introduce a hybrid data framework that combines MCTS with judgments from a strong VLM, producing more accurate step-level labels.
  arXiv  Detail & Related papers  (2025-09-27T10:56:58Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• Test-Time Learning for Large Language Models [33.11605667376906]
  We propose a Test-Time Learning (TTL) paradigm for Large Language Models (LLMs)<n>LLMs dynamically adapts to target domains using only unlabeled test data during testing.<n>We demonstrate through experiments that TLM improves performance by at least 20% compared to original LLMs on domain knowledge adaptation.
  arXiv  Detail & Related papers  (2025-05-27T02:18:59Z)
• Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought [58.321044666612174]
  Vad-R1 is an end-to-end MLLM-based framework for Video Anomaly Reasoning.<n>We design a Perception-to-Cognition Chain-of-Thought (P2C-CoT) that simulates the human process of recognizing anomalies.<n>We also propose an improved reinforcement learning algorithm AVA-GRPO, which explicitly incentivizes the anomaly reasoning capability of MLLMs.
  arXiv  Detail & Related papers  (2025-05-26T12:05:16Z)
• Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension [95.63899307791665]
  Vision Value Model (VisVM) can guide VLM inference-time search to generate responses with better visual comprehension.<n>In this paper, we present VisVM that can guide VLM inference-time search to generate responses with better visual comprehension.
  arXiv  Detail & Related papers  (2024-12-04T20:35:07Z)
• LOVM: Language-Only Vision Model Selection [13.857583570058392]
  We introduce a new task LOVM: Language-Only Vision Model Selection, where methods are expected to perform both model selection and performance prediction. We then introduced an extensive LOVM benchmark consisting of ground-truth evaluations of 35 pre-trained VLMs and 23 datasets.
  arXiv  Detail & Related papers  (2023-06-15T06:53:05Z)
• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models [76.410400238974]
  We propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. A CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. The proposed textitreinforcement learning with CLIP feedback(RLCF) framework is highly flexible and universal.
  arXiv  Detail & Related papers  (2023-05-29T11:03:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.