Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark

• URL: http://arxiv.org/abs/2607.21685v1
• Date: Thu, 23 Jul 2026 14:37:23 GMT
• Title: Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark
• Abstract summary: We use the Cohen et al. (2006) drug-class benchmark on three topics to examine how the Statins result changes under alternative designs.<n>In screening pipelines using transformers or 10-fold bag-of-words, the gap on the topics tested is about 0.02 WSS@95%, with CIs spanning zero on at least one bound.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A systematic review begins with someone reading thousands of abstracts to identify the few that are relevant, and classifiers are used to prioritise that reading. Their inputs are often augmented with Medical Subject Headings (MeSH), assigned either by expert indexers weeks or months after publication or by automatic tools at once. To our knowledge the two have not been compared directly as classifier features, and no previous work has asked whether that comparison's outcome depends on how the classifier is evaluated. Using the Cohen et al. (2006) drug-class benchmark on three topics, we characterise a bag-of-words logistic regression classifier (seven reruns) and BiomedBERT (five seeds), then examine how the Statins result changes under alternative designs. Under the canonical 5-fold full-corpus design, the bag-of-words expert-vs-auto gap on Statins is +0.096 WSS@95%. Matching the corpus size to the smaller topics (n = 803) reduces it to +0.033 (95% bootstrap CI includes zero), and 10-fold cross-validation at full size to +0.021 (CI narrowly excludes zero). Under canonical evaluation BiomedBERT gives +0.020, within sampling noise of the bag-of-words 10-fold result. A power analysis indicates a Statins-sized effect would not have been detectable at the Opioids or ADHD variance, so those nulls are design-limited rather than informative. A representation asymmetry remains: 15.1% of Statins inputs exceed BiomedBERT's 512-token limit when expert MeSH terms are appended, so truncation may contribute to the smaller transformer gap, although this cannot be separated from training volume here. In screening pipelines using transformers or 10-fold bag-of-words, the gap on the topics tested is about 0.02 WSS@95%, with CIs spanning zero on at least one bound. More broadly, benchmark conclusions about feature sources can change substantially under reasonable changes to the evaluation design.

Related papers

• Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review [2.9152186617858304]
  We study 11,429 reviews paired with 10,104 human-authored inline comments from AI.<n>Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods.<n>Four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles.
  arXiv  Detail & Related papers  (2026-09-05T18:15:18Z)
• Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence [0.7340017786387767]
  We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values.<n>On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label.
  arXiv  Detail & Related papers  (2026-08-26T14:41:24Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• Significance-First Splitting: Aligning Treatment Heterogeneity Detection with Honest Estimation [0.0]
  Estimating heterogeneous treatment effects (CATE) requires simultaneously detecting effect modification and quantifying estimation uncertainty.<n>Existing tree-based methods make an uneasy trade-off: significance-based approaches identify subgroup interactions directly but lack valid inference.<n>We introduce a hybrid algorithm that fuses significance-based splitting with honest sample-splitting and cross-validation.
  arXiv  Detail & Related papers  (2026-07-04T19:50:35Z)
• Evaluating Factual Density in Multi-Source RAG: A Study in Medical AI Accuracy [0.0]
  This paper introduces Factual Density (FD*), a novel retrieval optimization signal.<n>It measures the proportion of verified atomic claims relative to total token count.<n>FD* was the only condition to achieve 100% systematic review saturation in top-5 results.
  arXiv  Detail & Related papers  (2026-05-29T16:25:39Z)
• Reliable Extraction of Clinical Follow-Up Instructions: A Hybrid Neural-Symbolic Pipeline [43.0484058393522]
  We test a hybrid neural-symbolic pipeline against direct generation.<n>We evaluate on a 2,000-note synthetic outpatient corpus with action-disjoint splits.
  arXiv  Detail & Related papers  (2026-05-26T05:14:33Z)
• From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks [41.99844472131922]
  We provide a benchmark calibration for foundation-model weak labels.<n>The crossover predicted by theory appears at ng100 on PCAM, 20-50 on ISIC, and 250-500 on NIH-CXR.<n>A structured-vs-random noise sign flip on NIH-CXR shows that the rate-only formulation of the bound is incomplete.
  arXiv  Detail & Related papers  (2026-05-23T23:15:07Z)
• When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classification [0.0]
  It is shown that the claimed improvements in micro F1 for BERT-for-Patents mainly reflect a volume effect.<n>A corpus that allows for improvement in classification performance up to +0.58 in raw micro F1 may adversely affect a Jaccard-overlap retrieval proxy.
  arXiv  Detail & Related papers  (2026-05-22T23:49:52Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Representation Before Training: A Fixed-Budget Benchmark for Generative Medical Event Models [0.3250525349446657]
  We evaluate how representation decisions affect downstream prediction after a shared one-epoch pretraining budget.<n>We train 28 matched transformers on MIMIC-IV and evaluate them on 30 clinical outcomes.
  arXiv  Detail & Related papers  (2026-04-18T01:38:47Z)
• Identifying Evidence-Based Nudges in Biomedical Literature with Large Language Models [2.2015514798912412]
  We present a scalable, AI-powered system that identifies and extracts evidence-based behavioral nudges from unstructured biomedical literature.<n>Nudges are subtle, non-coercive interventions that influence behavior without limiting choice, showing strong impact on health outcomes like medication adherence.
  arXiv  Detail & Related papers  (2026-02-10T22:36:07Z)
• In Search of Lost Online Test-time Adaptation: A Survey [40.68806005826287]
  This article presents a comprehensive survey of online test-time adaptation (OTTA) We classify OTTA techniques into three primary categories and benchmark them using a modern backbone, the Vision Transformer (ViT) Our findings diverge from existing literature, revealing that transformers demonstrate heightened resilience to diverse domain shifts.
  arXiv  Detail & Related papers  (2023-10-31T05:47:33Z)
• Smoothed Contrastive Learning for Unsupervised Sentence Embedding [41.09180639504244]
  We introduce a smoothing strategy upon the InfoNCE loss function, termedGaussian Smoothing InfoNCE (GS-InfoNCE) GS-InfoNCE outperforms the state-of-the-art unsup-SimCSE by an average Spear-man correlation of 1.38%, 0.72%, 1.17% and 0.28% on the base of BERT-base, BERT-large,RoBERTa-base and RoBERTa-large, respectively.
  arXiv  Detail & Related papers  (2021-09-09T14:54:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.