Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

• URL: http://arxiv.org/abs/2604.04500v1
• Date: Mon, 06 Apr 2026 07:51:59 GMT
• Title: Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
• Abstract summary: We propose Saliency-R1, a framework for improving the interpretability and faithfulness of vision-language models (VLMs)<n>We introduce a novel saliency map technique that efficiently highlights critical image regions contributing to generated tokens without additional computational overhead.<n>Experiments show Saliency-R1 improves reasoning faithfulness, interpretability, and overall task performance.
• Score: 26.150136674969605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) have achieved remarkable success across diverse tasks. However, concerns about their trustworthiness persist, particularly regarding tendencies to lean more on textual cues than visual evidence and the risk of producing ungrounded or fabricated responses. To address these issues, we propose Saliency-R1, a framework for improving the interpretability and faithfulness of VLMs reasoning. Specifically, we introduce a novel saliency map technique that efficiently highlights critical image regions contributing to generated tokens without additional computational overhead. This can further be extended to trace how visual information flows through the reasoning process to the final answers, revealing the alignment between the thinking process and the visual context. We use the overlap between the saliency maps and human-annotated bounding boxes as the reward function, and apply Group Relative Policy Optimization (GRPO) to align the salient parts and critical regions, encouraging models to focus on relevant areas when conduct reasoning. Experiments show Saliency-R1 improves reasoning faithfulness, interpretability, and overall task performance.

Related papers

• iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning [35.83366559347647]
  Visually grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance fine-grained perception in large language models (MLLMs)<n>Mandating explicit object boxes in visually grounded CoT during inference often degrades performance compared to standard textual CoT.<n>We propose Internalizing Visually Grounded Reasoning (textbfiVGR), a novel reinforcement learning framework that transfers localization capabilities into the textual reasoning process.
  arXiv  Detail & Related papers  (2026-05-29T10:07:25Z)
• Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention [41.546578522790114]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for advancing complex reasoning in large language models.<n>We propose Faithful-MR1, a training framework that anchors and reinforces visual attention to address both halves of faithful multimodal reasoning.
  arXiv  Detail & Related papers  (2026-05-21T07:10:18Z)
• CAVE: A Structured Credit Assignment Approach for Fragmented Visual Evidence Reasoning [13.024838183478858]
  We propose Credit Assignment for Visual Evidence (CAVE) for interleaved visual reasoning.<n>CAVE evaluates the contribution of intermediate steps at the action level via three complementary reasoning process signals.<n>We construct TRACER-Bench, which covers four nonlocal and semantically confusable reasoning dimensions.
  arXiv  Detail & Related papers  (2026-05-13T16:50:24Z)
• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• ProFocus: Proactive Perception and Focused Reasoning in Vision-and-Language Navigation [57.65093237897077]
  We propose a training-free progressive framework that unifies underlineProactive Perception and underlineFocused Reasoning.<n>For proactive perception, ProFocus transforms panoramic observations into structured ego-centric semantic maps.<n>For focused reasoning, we propose Branch-Diverse Monte Carlo Tree Search (BD-MCTS) to identify top-$k$ high-value waypoints.
  arXiv  Detail & Related papers  (2026-03-01T04:13:18Z)
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing [62.447497430479174]
  Drawing to reason in space is a novel paradigm that enables LVLMs to reason through elementary drawing operations in the visual space.<n>Our model, named VILASR, consistently outperforms existing methods across diverse spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T17:41:50Z)
• Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning [96.01617809845396]
  Ground-R1 is a reinforcement learning framework that enables grounded visual reasoning without requiring explicit evidence or rationale annotations.<n>Ground-R1 achieves superior performance and exhibits emergent cognitive behaviors such as uncertainty awareness, spatial perception, and iterative refinement.
  arXiv  Detail & Related papers  (2025-05-26T17:51:47Z)
• VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought [51.43082554363725]
  We introduce textbfVLM-R$3$ (textbfVisual textbfLanguage textbfModel with textbfRegion textbfRecognition and textbfReasoning), a framework that equips an MLLM with the ability to decide emph when additional visual evidence is needed.<n>Experiments on MathVista, ScienceQA, and other benchmarks show that VLM-R$3$ sets a new
  arXiv  Detail & Related papers  (2025-05-22T03:50:13Z)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [69.56484419619919]
  We study the spatial reasoning challenge from the lens of mechanistic interpretability.<n>We observe that successful spatial reasoning correlates strongly with the model's ability to align its attention with actual object locations.<n>Motivated by these findings, we propose ADAPTVIS to sharpen the attention on highly relevant regions when confident.
  arXiv  Detail & Related papers  (2025-03-03T17:57:03Z)
• Improving Vision-and-Language Reasoning via Spatial Relations Modeling [30.477235227733928]
  Visual commonsense reasoning (VCR) is a challenging multi-modal task. The proposed method can guide the representations to maintain more spatial context. We achieve the state-of-the-art results on VCR and two other vision-and-language reasoning tasks VQA, and NLVR.
  arXiv  Detail & Related papers  (2023-11-09T11:54:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.