Reliable Benchmarking of Artifact Detection in Computational Pathology: A Reproducibility and Uncertainty Analysis

• URL: http://arxiv.org/abs/2608.30835v1
• Date: Mon, 31 Aug 2026 14:07:15 GMT
• Title: Reliable Benchmarking of Artifact Detection in Computational Pathology: A Reproducibility and Uncertainty Analysis
• Abstract summary: Methods: The protocol quantifies four sources of variability - test-set sampling, trainingity, partition composition, and undocumented preprocessing.<n>We apply it to an independent reconstruction of a published diffusion-based artifact detector, evaluated on the original 24-slide partition and against a supervised baseline.<n>Results: The method's central mechanism reproduces: the auxiliary contrastive term improves pooled F1 from 0.673 to 0.688 and replicates under a second seed.
• Score: 1.8864137667201766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Background and Objective: Quality control is a prerequisite for whole-slide image analysis, yet the benchmarks on which quality-control methods are compared share four properties that make their reported differences hard to interpret: few independent slides, annotation concentrated in a minority of them, pooled ratio metrics with no closed-form standard error, and a single inherited train/test partition. We propose a reliability protocol for such benchmarks. Methods: The protocol quantifies four sources of variability - test-set sampling, training stochasticity, partition composition, and undocumented preprocessing - a claim is reportable only if it survives all four; three of the four cost minutes of compute. We apply it to an independent reconstruction of a published diffusion-based artifact detector, evaluated on the original 24-slide partition and against a supervised baseline. Results: The method's central mechanism reproduces: the auxiliary contrastive term improves pooled F1 from 0.673 to 0.688 and replicates under a second seed (+0.0156, p = 0.031; +0.0190, p = 0.005), although it acts on pen marking rather than the artifact types cited to motivate it. Its comparative claims do not: differences between design variants, and against the supervised baseline, fall inside the uncertainty of the evaluation. Four of 24 slides carry 70% of scored annotated pixels, giving an effective sample size of 6.2, and the inherited partition sits at the 7th percentile. An unreported tissue-restriction step excludes 41.4% of out-of-focus annotation against 2.6% of air bubble; such a gate is confounded with blur by construction. Conclusions: Small-cohort benchmarks support far weaker conclusions than current reporting implies. The four checks are cheap enough to accompany any evaluation on such a resource and separate reproducible effects from differences the evaluation cannot resolve.

Related papers

• Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor [3.0704370207786265]
  We show that a counterfactual fairness estimate reported without a per-action floor beside it cannot be read as evidence of disparity.<n>The measurements were taken with FairMedAgent, an evaluation harness for disparity in the actions of clinical adjudication agents.
  arXiv  Detail & Related papers  (2026-09-02T23:38:17Z)
• Decomposing Wrong-Consensus Agreement in LLM Self-Consistency: A GPT-4.1 Case Study [1.2716872085463884]
  Majority voting over multiple LLM samples is widely used to raise answer accuracy, yet its gain varies erratically.<n>This paper gives a quantitative account of this failure.
  arXiv  Detail & Related papers  (2026-08-19T10:50:15Z)
• Automated binary classification of hazelnut X-ray images: A deep-learning benchmark for quality assessment [33.72751145910978]
  Non-destructive X-ray imaging can reveal internal hazelnut defects that are difficult to detect by external inspection alone.<n>We present a benchmark for binary hazelnut quality classification based on 799 segmented single- Kernel X-ray images.
  arXiv  Detail & Related papers  (2026-08-12T07:55:22Z)
• Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel [0.0]
  We study a fixed panel of sixteen lightweight persona-conditioned GPT-4.1 configurations in repeated strategic games.<n>Variation was strongly prompt-indexed, but its share depended on uncertainty assumptions.<n>Results concern one fixed model-prompt panel and do not establish human substitutability.
  arXiv  Detail & Related papers  (2026-08-02T04:10:11Z)
• Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity [43.23157850664282]
  We introduce VOIR DIRE, a multimodal benchmark of 626 culturally paired image-prompt artifacts spanning U.S. and mainland Chinese contexts.<n>Across six MLLMs, the bias decomposes into two failures: a positivity-floor calibration failure and an orientation failure.<n>We recommend reporting alignment against each reference pool separately and treating cross-pool divergence as a judge property.
  arXiv  Detail & Related papers  (2026-06-12T15:53:40Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Data [0.0]
  We introduce PAIR-CI, a nonparametric conditional independence (CI) test that restores calibration by integrating multiple imputation directly into the inferential procedure.<n>A provably consistent variance estimator jointly accounts for uncertainty arising from cross-validation and multiple imputation.
  arXiv  Detail & Related papers  (2026-05-06T12:34:37Z)
• When Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent Reasoning [16.505918019260964]
  We demonstrate that state-of-the-art models (Qwen2.5-Math-7B) achieve 61% accuracy through a mixture of reliable and unreliable predictions.<n>We show that 18.4% of correct predictions employ stable, faithful reasoning while 81.6% emerge through computationally inconsistent pathways.
  arXiv  Detail & Related papers  (2026-03-03T19:43:36Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Chest x-ray automated triage: a semiologic approach designed for clinical implementation, exploiting different types of labels through a combination of four Deep Learning architectures [83.48996461770017]
  This work presents a Deep Learning method based on the late fusion of different convolutional architectures. We built four training datasets combining images from public chest x-ray datasets and our institutional archive. We trained four different Deep Learning architectures and combined their outputs with a late fusion strategy, obtaining a unified tool.
  arXiv  Detail & Related papers  (2020-12-23T14:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.