mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health

• URL: http://arxiv.org/abs/2606.29467v1
• Date: Sun, 28 Jun 2026 15:51:53 GMT
• Title: mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
• Abstract summary: mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources.<n> mamaretrieval pairs 3,185 clinical queries with graded (0-6) relevance labels over a 63,650-chunk maternal-health guideline corpus.
• Score: 9.0025190634417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline retrieval. We release two benchmarks that fill these gaps. mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks; to help users calibrate the LLM judge that scores the rubric track, we re-scope HealthBench's physician-labelled meta-evaluation to the domain. mamaretrieval pairs 3,185 clinical queries with graded (0-6) relevance labels over a 63,650-chunk maternal-health guideline corpus, using a decomposed rubric that distinguishes a chunk that answers a query from one merely on its topic. Three decisions shape both: assemble and filter expert sources rather than author questions, grade relevance rather than binarise it, and measure and disclose the limits of the labels -- scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit -- rather than treat them as an oracle. A companion paper uses the benchmarks to evaluate a deployed on-device assistant; both are released openly for research.

Related papers

• MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification [50.13873960887466]
  We introduce MedSNIP-Bench, a benchmark for snippet-level medical fact verification.<n>We also introduce MedSNIP, an automatic snippet-generation pipeline.<n>Across MedSNIP-Bench, HealthFC, and MedHallu, snippet-level verification preserves or improves false-class F1.
  arXiv  Detail & Related papers  (2026-09-11T14:08:44Z)
• MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation [0.49454229616760287]
  Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation.<n>We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI.<n>We propose Evidence-Anchored RAG, a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective.
  arXiv  Detail & Related papers  (2026-07-15T21:56:19Z)
• MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar [9.0025190634417]
  We present MAM-AI, a question-answering assistant for nurse-midwives in Zanzibar.<n>A question is embedded (EmbeddingGemma, 300M) and matched against a curated corpus of 87 guideline documents.<n>A 4B int4 generator (Gemma 4 E4B) is fully offline, with no query leaving the device.
  arXiv  Detail & Related papers  (2026-06-28T19:52:04Z)
• QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models [8.050059911826338]
  Large Language Models (LLMs) excel on standardized medical exams, but high scores often fail to translate to high-quality responses for real-world medical queries.<n>We introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment.
  arXiv  Detail & Related papers  (2026-03-14T01:51:43Z)
• Developing and evaluating a chatbot to support maternal health care [20.297280325309924]
  We present a bot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital.<n>The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM.<n>Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision.
  arXiv  Detail & Related papers  (2026-03-13T17:02:05Z)
• MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Prompts [70.64143198545031]
  We propose MedREK, a retrieval-based editing framework that integrates a shared query-key module for precise matching with an attention-based prompt encoder for informative guidance.<n>Our results on various medical benchmarks demonstrate that our MedREK achieves superior performance across different core metrics.
  arXiv  Detail & Related papers  (2025-10-15T12:50:33Z)
• Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models [57.73472878679636]
  We introduce Med-RewardBench, the first benchmark specifically designed to evaluate medical reward models and judges.<n>Med-RewardBench features a multimodal dataset spanning 13 organ systems and 8 clinical departments, with 1,026 expert-annotated cases.<n>A rigorous three-step process ensures high-quality evaluation data across six clinically critical dimensions.
  arXiv  Detail & Related papers  (2025-08-29T08:58:39Z)
• PediatricsMQA: a Multi-modal Pediatrics Question Answering Benchmark [11.57757126151033]
  Large language models (LLMs) and vision-augmented LLMs (VLMs) have advanced medical informatics, diagnostics, and decision support.<n>These models exhibit systematic biases, particularly age bias, compromising their reliability and equity.<n>This bias reflects a broader imbalance in medical research, where pediatric studies receive less funding and representation.
  arXiv  Detail & Related papers  (2025-08-22T14:50:55Z)
• LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation [58.25892575437433]
  evaluating large language models (LLMs) in medicine is crucial because medical applications require high accuracy with little room for error.<n>We present LLMEval-Med, a new benchmark covering five core medical areas, including 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios.
  arXiv  Detail & Related papers  (2025-06-04T15:43:14Z)
• A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models [57.88111980149541]
  We introduce Asclepius, a novel Med-MLLM benchmark that assesses Med-MLLMs in terms of distinct medical specialties and different diagnostic capacities.<n>Grounded in 3 proposed core principles, Asclepius ensures a comprehensive evaluation by encompassing 15 medical specialties.<n>We also provide an in-depth analysis of 6 Med-MLLMs and compare them with 3 human specialists.
  arXiv  Detail & Related papers  (2024-02-17T08:04:23Z)
• Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation [58.54483567073125]
  We propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists. We observed good levels of inter-annotator agreement in a first evaluation study using the protocol.
  arXiv  Detail & Related papers  (2022-11-17T10:54:28Z)
• Self-supervised Answer Retrieval on Clinical Notes [68.87777592015402]
  We introduce CAPR, a rule-based self-supervision objective for training Transformer language models for domain-specific passage matching. We apply our objective in four Transformer-based architectures: Contextual Document Vectors, Bi-, Poly- and Cross-encoders. We report that CAPR outperforms strong baselines in the retrieval of domain-specific passages and effectively generalizes across rule-based and human-labeled passages.
  arXiv  Detail & Related papers  (2021-08-02T10:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.