ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answering

• URL: http://arxiv.org/abs/2608.10996v1
• Date: Tue, 11 Aug 2026 14:48:15 GMT
• Title: ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answering
• Abstract summary: Reinforcement learning with verifiable rewards has been especially effective in mathematics and coding.<n>We introduce ConRub-Med to preserve useful distinctions as rubric feedback moves from construction to policy optimization.<n>ConRub-Med ranks first on six of nine benchmarks and the highest medical and generalization averages.
• Score: 33.27644708906731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards has been especially effective in mathematics and coding, where answers can be checked automatically. Many open-ended medical questions lack comparably cheap outcome verifiers: responses may be partly correct, incomplete, or contain clinically consequential errors. Rubrics written or validated by physicians offer strong clinical grounding, but involving experts in every instance is costly. Model-generated rubrics make this supervision scalable. We introduce ConRub-Med to preserve useful distinctions as rubric feedback moves from construction to policy optimization. For each prompt, three heterogeneous language models propose atomic criteria independently; a separate model reviews them, retaining only criteria with semantic support from all three generators. Three-State scoring distinguishes correct coverage, missing information, and incorrect claims. Errors receive negative rather than zero credit. When every response in a complete Group Relative Policy Optimization (GRPO) group receives the same final reward, a pairwise judge provides sequence advantages only if both candidate orders agree, without changing the scalar rewards. Groups without ties use vanilla GRPO. In a blinded study matched by question, two medical experts rate panels from the full pipeline as more clinically relevant than panels produced by one generator. Across the evaluated open models, ConRub-Med ranks first on six of nine benchmarks and achieves the highest medical and generalization averages. Using the resulting rubric dataset of 5,166 prompts, it scores $38.98 \pm 1.04$ (mean $\pm$ SD) on HealthBench-Hard, compared with InfiMed-ORBIT's 33.60 with 8,000 samples and 37.30 with 28,000.

Related papers

• MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification [50.13873960887466]
  We introduce MedSNIP-Bench, a benchmark for snippet-level medical fact verification.<n>We also introduce MedSNIP, an automatic snippet-generation pipeline.<n>Across MedSNIP-Bench, HealthFC, and MedHallu, snippet-level verification preserves or improves false-class F1.
  arXiv  Detail & Related papers  (2026-09-11T14:08:44Z)
• Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks [0.028647133890966986]
  Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes.<n> Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative.<n>We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected.
  arXiv  Detail & Related papers  (2026-09-10T01:18:39Z)
• ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV [0.0]
  Reasoning benchmarks measure clinical performance on clean inputs.<n>We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one.<n>EpiKG carries assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent.
  arXiv  Detail & Related papers  (2026-05-11T18:47:52Z)
• MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence [24.517280048376758]
  medvigil is a 300-case evaluation suite drawn from four public medical VQA sources.<n>Every gold answer, refusal option, candidate-answer set, paraphrase, false-premise trap, ROI box, and clinical risk tier is clinician-authored.<n>We report seven correctness-conditioned audit metrics that summarise into the medvigil Composite Score (MCS)<n>The independent radiologist scores MCS 83.3 at silent-failure rate 5.8%, leaving a 14.1-point composite headroom above the strongest audited model.
  arXiv  Detail & Related papers  (2026-05-08T15:55:30Z)
• Ruling Out to Rule In: Contrastive Hypothesis Retrieval for Medical Question Answering [1.6886041345889138]
  We propose Contrastive Hypothesis Retrieval (CHR), a framework inspired by the process of clinical differential diagnosis.<n>CHR scores documents by promoting $H+$-aligned evidence while penalizing $H-$-aligned content.<n>CHR bridges clinical reasoning with retrieval mechanism design and offers a practical path to reducing hard-negative contamination in medical RAG systems.
  arXiv  Detail & Related papers  (2026-04-06T11:13:57Z)
• Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness [49.2667937337333]
  We stress-test this assumption for detecting incomplete patient-facing medical responses.<n>We evaluate three granularities (General-Likert, Analytical-Rubric, Dynamic-Checklist) and three backbone models across two clinician-annotated datasets.
  arXiv  Detail & Related papers  (2026-03-26T19:01:55Z)
• QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models [8.050059911826338]
  Large Language Models (LLMs) excel on standardized medical exams, but high scores often fail to translate to high-quality responses for real-world medical queries.<n>We introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment.
  arXiv  Detail & Related papers  (2026-03-14T01:51:43Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning [54.30630356786752]
  ReasonMed is the largest medical reasoning dataset to date, with 370k high-quality examples.<n>It is built through a multi-agent generation, verification, and refinement process.<n>Using ReasonMed, we find that integrating detailed CoT reasoning with concise answer summaries yields the most robust fine-tuning results.
  arXiv  Detail & Related papers  (2025-06-11T08:36:55Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.