Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark

• URL: http://arxiv.org/abs/2411.18831v2
• Date: Fri, 19 Sep 2025 20:50:29 GMT
• Title: Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark
• Abstract summary: We present a benchmark for measuring the methodological strength of biomedical papers.<n>We draw on the risk-of-bias framework used for systematic reviews.<n>Our analyses show that large language models' reasoning and retrieval capabilities impact their effectiveness with risk-of-bias assessment.
• Score: 5.549390587677274
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Systems that answer questions by reviewing the scientific literature are becoming increasingly feasible. To draw reliable conclusions, these systems should take into account the quality of available evidence from different studies, placing more weight on studies that use a valid methodology. We present a benchmark for measuring the methodological strength of biomedical papers, drawing on the risk-of-bias framework used for systematic reviews. Derived from over 500 biomedical studies, the three benchmark tasks encompass expert reviewers' judgments of studies' research methodologies, including the assessments of risk of bias within these studies. The benchmark contains a human-validated annotation pipeline for fine-grained alignment of reviewers' judgments with research paper sentences. Our analyses show that large language models' reasoning and retrieval capabilities impact their effectiveness with risk-of-bias assessment. The dataset is available at https://github.com/RoBBR-Benchmark/RoBBR.

Related papers

• Ontology-Guided Multi-Agent Extraction of Evaluation Objects from Academic Review Texts: Evidence from Chinese Library and Information Science [6.306930443447107]
  This study proposes an ontology-guided multi-agent framework for evaluation object extraction.<n>It achieves a Precision of 90.33%, Recall of 84.55%, Entity-level F1 of 87.34%, Strict Typed F1 of 79.78%, and Type Accuracy of 91.35%.
  arXiv  Detail & Related papers  (2026-08-30T03:09:47Z)
• Enhancing Scientific Literature Chatbots with Retrieval-Augmented Generation: A Performance Evaluation of Vector and Graph-Based Systems [1.0832844764942349]
  This paper investigates the enhancement of scientific literature through retrieval-augmented generation (RAG)<n>The proposed chatbots leverages both structured (graph) and unstructured (vector) databases to access scientific articles and gray literature.<n> Benchmark test sets were generated using a GPT model, with selected outputs annotated for evaluation.
  arXiv  Detail & Related papers  (2026-02-19T21:42:02Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• BABE: Biology Arena BEnchmark [51.53220868983288]
  BABE is a benchmark designed to evaluate the experimental reasoning capabilities of biological AI systems.<n>Our benchmark provides a robust framework for assessing how well AI systems can reason like practicing scientists.
  arXiv  Detail & Related papers  (2026-02-05T16:39:20Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature Tasks [87.29946641069068]
  We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature tasks.<n>By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks.<n>We release SciArena-Eval, a meta-evaluation benchmark based on our collected preference data.
  arXiv  Detail & Related papers  (2025-07-01T17:51:59Z)
• Can Large Language Models Match the Conclusions of Systematic Reviews? [43.27182445778988]
  We ask: Can large language models (LLMs) match the conclusions of systematic reviews written by clinical experts when given access to the same studies?<n>We benchmark 24 LLMs on MedEvidence, including reasoning, non-reasoning, medical specialist, and models across varying sizes (from 7B-700B).<n>We find that reasoning does not necessarily improve performance, larger models do not consistently yield greater gains, and knowledge-based fine-tuning degrades accuracy on MedEvidence.
  arXiv  Detail & Related papers  (2025-05-28T18:58:09Z)
• Uncertainty-aware abstention in medical diagnosis based on medical texts [87.88110503208016]
  This study addresses the critical issue of reliability for AI-assisted medical diagnosis.<n>We focus on the selection prediction approach that allows the diagnosis system to abstain from providing the decision if it is not confident in the diagnosis.<n>We introduce HUQ-2, a new state-of-the-art method for enhancing reliability in selective prediction tasks.
  arXiv  Detail & Related papers  (2025-02-25T10:15:21Z)
• Pub-Guard-LLM: Detecting Retracted Biomedical Articles with Reliable Explanations [11.082285990214595]
  Pub-Guard-LLM is a large language model-based system tailored to fraud detection of biomedical scientific articles.<n>Pub-Guard-LLM consistently surpasses the performance of various baselines.<n>By enhancing both detection performance and explainability in scientific fraud detection, Pub-Guard-LLM contributes to safeguarding research integrity with a novel, effective, open-source tool.
  arXiv  Detail & Related papers  (2025-02-21T12:54:56Z)
• exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem [11.763640675057076]
  We develop a benchmark dataset for evaluating the reviewer assignment problem without needing explicit labels. We benchmark various methods, including traditional lexical matching, static neural embeddings, and contextualized neural embeddings. Our results indicate that while traditional methods perform reasonably well, contextualized embeddings trained on scholarly literature show the best performance.
  arXiv  Detail & Related papers  (2025-02-11T16:35:04Z)
• Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation [0.0]
  This study presents a framework for automated evaluation of dynamically evolving topic in scientific literature using Large Language Models (LLMs) The proposed approach harnesses LLMs to measure key quality dimensions, such as coherence, repetitiveness, diversity, and topic-document alignment, without heavy reliance on expert annotators or narrow statistical metrics.
  arXiv  Detail & Related papers  (2025-02-11T08:23:56Z)
• Time Series Embedding Methods for Classification Tasks: A Review [2.8084422332394428]
  We present a comprehensive review and evaluation of time series embedding methods for effective representations in machine learning and deep learning models. We introduce a taxonomy of embedding techniques, categorizing them based on their theoretical foundations and application contexts. Our experimental results demonstrate that the performance of embedding methods varies significantly depending on the dataset and classification algorithm used.
  arXiv  Detail & Related papers  (2025-01-23T05:24:45Z)
• Benchmark for Evaluation and Analysis of Citation Recommendation Models [0.0]
  We develop a benchmark specifically designed to analyze and compare citation recommendation models. This benchmark will evaluate the performance of models on different features of the citation context. This will enable meaningful comparisons and help identify promising approaches for further research and development in the field.
  arXiv  Detail & Related papers  (2024-12-10T18:01:33Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [52.228708947607636]
  This paper introduces a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods. The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics. We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
  This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews. The newly emerging AI-generated literature reviews are also appraised. This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Dyport: Dynamic Importance-based Hypothesis Generation Benchmarking Technique [2.0077755400451855]
  This paper presents a novel benchmarking framework Dyport for evaluating biomedical hypothesis generation systems. We integrate knowledge from curated databases into a dynamic graph, accompanied by a method to quantify discovery importance. Being flexible, our benchmarking system is designed for broad application in hypothesis generation quality verification.
  arXiv  Detail & Related papers  (2023-12-06T06:07:50Z)
• Application of Transformers based methods in Electronic Medical Records: A Systematic Literature Review [77.34726150561087]
  This work presents a systematic literature review of state-of-the-art advances using transformer-based methods on electronic medical records (EMRs) in different NLP tasks.
  arXiv  Detail & Related papers  (2023-04-05T22:19:42Z)
• Towards Assessing Data Bias in Clinical Trials [0.0]
  Health care datasets can still be affected by data bias. Data bias provides a distorted view of reality, leading to wrong analysis results and, consequently, decisions. This paper proposes a method to address bias in datasets that: (i) defines the types of data bias that may be present in the dataset, (ii) characterizes and quantifies data bias with adequate metrics, and (iii) provides guidelines to identify, measure, and mitigate data bias for different data sources.
  arXiv  Detail & Related papers  (2022-12-19T17:10:06Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• A Comparative Study on Unsupervised Anomaly Detection for Time Series: Experiments and Analysis [28.79393419730138]
  Time series anomaly detection is often essential to enable reliability and safety. Many recent studies target anomaly detection for time series data. We introduce for data, methods, and evaluation strategies. We systematically evaluate and compare state-of-the-art traditional as well as deep learning techniques.
  arXiv  Detail & Related papers  (2022-09-10T10:44:25Z)
• On the role of benchmarking data sets and simulations in method comparison studies [0.0]
  This paper investigates differences and similarities between simulation studies and benchmarking studies. We borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.
  arXiv  Detail & Related papers  (2022-08-02T13:47:53Z)
• Best Practices and Scoring System on Reviewing A.I. based Medical Imaging Papers: Part 1 Classification [0.9428556282541211]
  The Machine Learning Education Sub-Committee of SIIM has identified a knowledge gap and a serious need to establish guidelines for reviewing these studies. This first entry in the series focuses on the task of image classification. The goal of this series is to provide resources to help improve the review process for A.I.-based medical imaging papers.
  arXiv  Detail & Related papers  (2022-02-03T21:46:59Z)
• Validating GAN-BioBERT: A Methodology For Assessing Reporting Trends In Clinical Trials [3.164363223464948]
  This study develops a sentiment classification algorithm for clinical trial abstracts using a semi-supervised natural language process model. The use of this algorithm was found to have a classification accuracy of 91.3%, with a macro F1-Score of 0.92, which is a significant improvement in accuracy when compared to previous methods.
  arXiv  Detail & Related papers  (2021-06-01T17:51:54Z)
• Quantitative Evaluations on Saliency Methods: An Experimental Study [6.290238942982972]
  We briefly summarize the status quo of the metrics, including faithfulness, localization, false-positives, sensitivity check, and stability. We conclude that among all the methods we compare, no single explanation method dominates others in all metrics.
  arXiv  Detail & Related papers  (2020-12-31T14:13:30Z)
• A Survey on Text Classification: From Shallow to Deep Learning [83.47804123133719]
  The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification.
  arXiv  Detail & Related papers  (2020-08-02T00:09:03Z)
• A Survey on Causal Inference [64.45536158710014]
  Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics. Various causal effect estimation methods for observational data have sprung up.
  arXiv  Detail & Related papers  (2020-02-05T21:35:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.