When to Rethink: Learning Multi-Perspective Self-Verification for Vision-Language Models
Abstract Overview
This paper studies self-verification for vision-language models (VLMs), focusing on when a model should trust its current answer and when it should rethink. The authors first analyze two factors that affect verification reliability: verifier capability and prompt design. Based on these findings, they propose MOTIVE, a multi-view self-verification framework that evaluates candidate answers through five complementary direct verification prompts, learns a correctness-aligned aggregated reliability score, and uses that score to make accept-or-rethink decisions. Experiments on five multimodal question answering benchmarks and three VLM backbones show that the approach improves both answer quality and the reliability of verification-driven reasoning.
Novelty
The paper's main novelty is a learned multi-view self-verification framework for VLMs that aggregates multiple prompt-based verification perspectives instead of relying on a single verification criterion. It explicitly trains an aggregated reliability score using view-level, aggregate-level, and calibration objectives to drive selective rethinking without requiring an external judge.
Results
Across five benchmarks and three backbones, MOTIVE consistently achieves the highest average accuracy, outperforming prompt-based self-verification baselines and multimodal self-improvement methods. It also improves accept-or-rethink decision quality as measured by Self-Verification Accuracy and reduces average reasoning turns under a maximum budget of three turns.
Key Points
- Preliminary analysis shows that verification quality depends on verifier capability and is highly sensitive to prompt design, with no single prompt consistently dominating across tasks.
- MOTIVE combines five direct verification views, averages their YES probabilities, and trains the resulting reliability estimate to align with candidate-answer correctness for selective rethinking.
- Empirical studies and ablations show that verification-view diversity, multi-level loss supervision, and confidence-preserving probability averaging contribute to higher final accuracy and better decision reliability.