Breaking Reward Collapse: Adaptive Reinforcement for Open-ended Medical Reasoning with Enhanced Semantic Discrimination

• URL: http://arxiv.org/abs/2508.12957v1
• Date: Mon, 18 Aug 2025 14:31:26 GMT
• Title: Breaking Reward Collapse: Adaptive Reinforcement for Open-ended Medical Reasoning with Enhanced Semantic Discrimination
• Abstract summary: Reinforcement learning with rule-based rewards has demonstrated strong potential in enhancing the reasoning and generalization capabilities of vision-language models (VLMs) and large language models (LLMs)<n>Existing reinforcement fine-tuning (RFT) approaches in this domain primarily target closed-ended visual question answering (VQA)<n>We propose ARMed, a novel RL framework for open-ended medical VQA.
• Score: 5.685365519519041
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement learning (RL) with rule-based rewards has demonstrated strong potential in enhancing the reasoning and generalization capabilities of vision-language models (VLMs) and large language models (LLMs), while reducing computational overhead. However, its application in medical imaging remains underexplored. Existing reinforcement fine-tuning (RFT) approaches in this domain primarily target closed-ended visual question answering (VQA), limiting their applicability to real-world clinical reasoning. In contrast, open-ended medical VQA better reflects clinical practice but has received limited attention. While some efforts have sought to unify both formats via semantically guided RL, we observe that model-based semantic rewards often suffer from reward collapse, where responses with significant semantic differences receive similar scores. To address this, we propose ARMed (Adaptive Reinforcement for Medical Reasoning), a novel RL framework for open-ended medical VQA. ARMed first incorporates domain knowledge through supervised fine-tuning (SFT) on chain-of-thought data, then applies reinforcement learning with textual correctness and adaptive semantic rewards to enhance reasoning quality. We evaluate ARMed on six challenging medical VQA benchmarks. Results show that ARMed consistently boosts both accuracy and generalization, achieving a 32.64% improvement on in-domain tasks and an 11.65% gain on out-of-domain benchmarks. These results highlight the critical role of reward discriminability in medical RL and the promise of semantically guided rewards for enabling robust and clinically meaningful multimodal reasoning.

Related papers

• AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes [59.134503227559634]
  Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding.<n>Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language.<n>We propose Affect Omni, a framework for verifiable affective reasoning.
  arXiv  Detail & Related papers  (2026-08-24T11:25:04Z)
• MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering [8.481971263315854]
  textscMedLVR introduces an explicit visual evidence state into autoretext decoding.<n>Latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence.
  arXiv  Detail & Related papers  (2026-04-10T16:03:03Z)
• MediX-R1: Open Ended Medical Reinforcement Learning [30.11159628872015]
  We introduce MediX-R1, an open-ended Reinforcement Learning framework for medical multimodal large language models (MLLMs)<n> MediX-R1 fine-tunes a baseline vision- backbone with Group Based RL and a composite reward tailored for medical reasoning.<n>Our results demonstrate that open-ended RL with comprehensive reward signals and LLM-based evaluation is a practical path toward reliable medical reasoning in multimodal models.
  arXiv  Detail & Related papers  (2026-02-26T18:59:46Z)
• R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging [69.96389360650072]
  We show that reasoning fidelity is highly predictive of downstream RLHF outcomes, beyond standard label accuracy.<n>We propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment.
  arXiv  Detail & Related papers  (2026-02-06T15:17:11Z)
• From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAG [65.0203623486525]
  Large Language Models (LLMs) exhibit high reasoning capacity in medical question-answering.<n>Their tendency to produce hallucinations and outdated knowledge poses critical risks in healthcare fields.<n>Existing methods rely on noisy token-level signals and lack the multi-round refinement required for complex reasoning.
  arXiv  Detail & Related papers  (2026-02-06T08:25:30Z)
• S-Chain: Structured Visual Chain-of-Thought For Medicine [81.97605645734741]
  We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT)<n>The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability.<n>S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical vision-language models.
  arXiv  Detail & Related papers  (2025-10-26T15:57:14Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• MedFact-R1: Towards Factual Medical Reasoning via Pseudo-Label Augmentation [25.186622292311228]
  MEDFACT-R1 is a two-stage framework that integrates external knowledge grounding with reinforcement learning.<n>It delivers up to 22.5% absolute improvement in factual accuracy over previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-09-18T16:59:59Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)
• GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via Reinforcement Learning [50.94508930739623]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability, impairing the ability of clinicians and patients to understand and trust model-generated answers.<n>This work first proposes a Thinking with Visual Grounding dataset wherein the answer generation is decomposed into intermediate reasoning steps.<n>We introduce a novel verifiable reward mechanism for reinforcement learning to guide post-training, improving the alignment between the model's reasoning process and its final answer.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• CAPO: Reinforcing Consistent Reasoning in Medical Decision-Making [42.28216499263317]
  We introduce Med-Zero-17K, a curated dataset for pure RL-based training, encompassing over 30 medical image modalities and 24 clinical tasks.<n>We propose a novel large-scale RL framework for Med-VLMs, which integrates rewards to ensure fidelity between perception and reasoning, consistency in reasoning-to-answer, and rule-based accuracy for final responses.
  arXiv  Detail & Related papers  (2025-06-15T13:42:46Z)
• Med-REFL: Medical Reasoning Enhancement via Self-Corrected Fine-grained Reflection [7.584796006142439]
  We propose Med-REFL, a underlinetextbfMedical underlinetextbfReasoning underlinetextbfEnhancement via self-corrected underlinetextbfFine-grained refunderlinetextbfLection.<n>Our method leverages a tree-of-thought approach to decompose medical questions into fine-grained reasoning paths, quantitatively evaluating each step and its subsequent reflections.
  arXiv  Detail & Related papers  (2025-06-11T14:58:38Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)
• Improving Medical Reasoning with Curriculum-Aware Reinforcement Learning [2.262453679768892]
  We introduce textbfMedCCO, the first multimodal reinforcement learning framework tailored for medical VQA.<n>MedCCO is fine-tuned on a diverse set of close-ended medical VQA tasks to establish domain-grounded reasoning capabilities.<n>We validate MedCCO across eight challenging medical VQA benchmarks, spanning both close-ended and open-ended settings.
  arXiv  Detail & Related papers  (2025-05-25T16:20:55Z)
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains [92.36624674516553]
  Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs)<n>We investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education.<n>We utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications.
  arXiv  Detail & Related papers  (2025-03-31T08:22:49Z)
• Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models [6.176432104264649]
  Vision-language models (VLMs) have achieved impressive progress in natural image reasoning, yet their potential in medical imaging remains underexplored.<n>We propose Med-R1, a reinforcement learning (RL)-enhanced vision-language model designed to improve generalization and reliability in medical reasoning.<n>We evaluate Med-R1 across eight distinct medical imaging modalities.
  arXiv  Detail & Related papers  (2025-03-18T06:12:38Z)
• MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning [29.84956540178252]
  Reasoning is a critical frontier for advancing medical image analysis.<n>We introduce MedVLM-R1, a medical VLM that explicitly generates natural language reasoning.<n>MedVLM-R1 boosts accuracy from 55.11% to 78.22% across MRI, CT, and X-ray benchmarks.
  arXiv  Detail & Related papers  (2025-02-26T23:57:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.