EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers

• URL: http://arxiv.org/abs/2504.18736v2
• Date: Fri, 08 Aug 2025 01:40:07 GMT
• Title: EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers
• Abstract summary: We study the task of automatically finding evidence relevant to hypotheses in biomedical papers.<n>We introduce EvidenceBench to measure models performance on this task.<n>We demonstrate the pipeline's validity and accuracy with multiple sets of human-expert annotations.
• Score: 6.016315914361666
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We study the task of automatically finding evidence relevant to hypotheses in biomedical papers. Finding relevant evidence is an important step when researchers investigate scientific hypotheses. We introduce EvidenceBench to measure models performance on this task, which is created by a novel pipeline that consists of hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence, completely guided by and faithfully following existing human experts judgment. We demonstrate the pipeline's validity and accuracy with multiple sets of human-expert annotations. We evaluated a diverse set of language models and retrieval systems on the benchmark and found that model performances still fall significantly short of the expert level on this task. To show the scalability of our proposed pipeline, we create a larger EvidenceBench-100k with 107,461 fully annotated papers with hypotheses to facilitate model training and development. Both datasets are available at https://github.com/EvidenceBench/EvidenceBench

Related papers

• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• PlantMarkerBench: A Multi-Species Benchmark for Evidence-Grounded Plant Marker Reasoning [10.873172077806798]
  PlantMarkerBench is a benchmark for evaluating literature-grounded plant marker evidence interpretation from full-text biological papers.<n>It spans four plant species and contains 5,550 sentence-level evidence instances annotated for marker-evidence validity, evidence type, and support strength.<n>We benchmark diverse open-weight and closed-source language models across species and prompting strategies.
  arXiv  Detail & Related papers  (2026-05-11T05:57:15Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases [10.00386797940562]
  We present FlyBench to evaluate AI agents on end-to-end agentic curation from scientific literature.<n>Given only a gene symbol, agents must search and read from a corpus of 16,898 full-text papers to produce structured annotations.<n>The benchmark includes 7,397 expert-curated annotations across 100 genes drawn from FlyBase.
  arXiv  Detail & Related papers  (2026-02-09T20:12:38Z)
• Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models [9.664217498808338]
  We introduce a unified taxonomy of evidence-based text generation with large language models.<n>We investigate 300 evaluation metrics across seven key dimensions.<n>We highlight open challenges and outline promising directions for future work.
  arXiv  Detail & Related papers  (2025-08-21T09:36:35Z)
• BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation [0.0]
  Existing automated methods struggle to generate novel and evidence-grounded hypotheses.<n>BioDisco is a multi-agent framework that draws upon language model-based reasoning and a dual-mode evidence system.
  arXiv  Detail & Related papers  (2025-08-02T09:32:52Z)
• BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research [29.469867701731374]
  BioDSA-1K consists of 1,029 hypothesis-centric tasks paired with 1,177 analysis plans.<n>The benchmark enables evaluation along four axes: (1) hypothesis decision accuracy, (2) alignment between evidence and conclusion, (3) correctness of the reasoning process, and (4) executability of the AI-generated analysis code.
  arXiv  Detail & Related papers  (2025-05-22T01:02:21Z)
• Unstructured Evidence Attribution for Long Context Query Focused Summarization [53.08341620504465]
  We propose to extract unstructured (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case.<n>We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle"
  arXiv  Detail & Related papers  (2025-02-20T09:57:42Z)
• Retrieving Versus Understanding Extractive Evidence in Few-Shot Learning [4.230202411425062]
  We analyze the relationship between the retrieval and interpretation of within-document evidence for large language model.<n>We perform two ablation studies to investigate when both label prediction and evidence retrieval errors can be attributed to qualities of the relevant evidence.
  arXiv  Detail & Related papers  (2025-02-19T20:48:09Z)
• Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark [2.0605929006983454]
  We present a benchmark for measuring the methodological strength of biomedical papers.<n>The benchmark tasks, drawn from more than 500 papers, cover the analysis of research study methodology, followed by evaluation of risk of bias.<n>We evaluate a range of large language models on the benchmark, and find that these models fall significantly short of expert-level performance.
  arXiv  Detail & Related papers  (2024-11-28T00:21:31Z)
• Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that the pointwise mutual information between a context and a question is an effective gauge for language model performance. We propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Causal Representation Learning from Multimodal Biomedical Observations [57.00712157758845]
  We develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets.<n>Key theoretical contribution is the structural sparsity of causal connections between modalities.<n>Results on a real-world human phenotype dataset are consistent with established biomedical research.
  arXiv  Detail & Related papers  (2024-11-10T16:40:27Z)
• GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction [15.246183329778656]
  Document-level relation extraction (DocRE) aims to extract relations between entities from unstructured document text. To overcome these challenges, we propose GEGA, a novel model for DocRE. We evaluate the GEGA model on three widely used benchmark datasets: DocRED, Re-DocRED, and Revisit-DocRED.
  arXiv  Detail & Related papers  (2024-07-31T07:15:33Z)
• Dyport: Dynamic Importance-based Hypothesis Generation Benchmarking Technique [2.0077755400451855]
  This paper presents a novel benchmarking framework Dyport for evaluating biomedical hypothesis generation systems. We integrate knowledge from curated databases into a dynamic graph, accompanied by a method to quantify discovery importance. Being flexible, our benchmarking system is designed for broad application in hypothesis generation quality verification.
  arXiv  Detail & Related papers  (2023-12-06T06:07:50Z)
• Complex Claim Verification with Evidence Retrieved in the Wild [73.19998942259073]
  We present the first fully automated pipeline to check real-world claims by retrieving raw evidence from the web. Our pipeline includes five components: claim decomposition, raw document retrieval, fine-grained evidence retrieval, claim-focused summarization, and veracity judgment.
  arXiv  Detail & Related papers  (2023-05-19T17:49:19Z)
• A Multi-Level Attention Model for Evidence-Based Fact Checking [58.95413968110558]
  We present a simple model that can be trained on sequence structures. Results on a large-scale dataset for Fact Extraction and VERification show that our model outperforms the graph-based approaches.
  arXiv  Detail & Related papers  (2021-06-02T05:40:12Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)
• Weakly- and Semi-supervised Evidence Extraction [107.47661281843232]
  We propose new methods to combine few evidence annotations with abundant document-level labels for the task of evidence extraction. Our approach yields substantial gains with as few as hundred evidence annotations.
  arXiv  Detail & Related papers  (2020-11-03T04:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.