Data Leakage and Redundancy in the LIT-PCBA Benchmark

• URL: http://arxiv.org/abs/2507.21404v2
• Date: Thu, 07 Aug 2025 01:10:51 GMT
• Title: Data Leakage and Redundancy in the LIT-PCBA Benchmark
• Abstract summary: LIT-PCBA is widely used to benchmark virtual screening models.<n>We find extensive data leakage and molecular redundancy across its models.<n>Almost all published results on LIT-PCBA are undermined.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LIT-PCBA is widely used to benchmark virtual screening models, but our audit reveals that it is fundamentally compromised. We find extensive data leakage and molecular redundancy across its splits, including 2D-identical ligands within and across partitions, pervasive analog overlap, and low-diversity query sets. In ALDH1 alone, for instance, 323 active training -- validation analog pairs occur at ECFP4 Tanimoto similarity $\geq 0.6$; across all targets, 2,491 2D-identical inactives appear in both training and validation, with very few corresponding actives. These overlaps allow models to succeed through scaffold memorization rather than generalization, inflating enrichment factors and AUROC scores. These flaws are not incidental -- they are so severe that a trivial memorization-based baseline with no learnable parameters can exploit them to match or exceed the reported performance of state-of-the-art deep learning and 3D-similarity models. As a result, nearly all published results on LIT-PCBA are undermined. Even models evaluated in "zero-shot" mode are affected by analog leakage into the query set, weakening claims of generalization. In its current form, the benchmark does not measure a model's ability to recover novel chemotypes and should not be taken as evidence of methodological progress. All code, data, and baseline implementations are available at: https://github.com/sievestack/LIT-PCBA-audit

Related papers

• When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion [0.0]
  Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.<n>We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$.<n>The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups.
  arXiv  Detail & Related papers  (2026-07-12T17:17:18Z)
• Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks [1.9800951131982487]
  We use Speculative Refinement (SpecRef) to warm-start a masked diffusion language model from an AR draft.<n>We evaluate SpecRef across six benchmarks (HumanEval, MBPP, GSM8K, BBH, ARC-Challenge, HellaSwag) with three distinct evaluation protocols.<n>These observations apply to any multi-stage or non-autoregressive generation pipeline and point toward more diagnostic evaluation practices.
  arXiv  Detail & Related papers  (2026-06-25T18:52:24Z)
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing [29.035258104995204]
  We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
  arXiv  Detail & Related papers  (2026-05-13T18:01:05Z)
• TabSHAP [5.3259362673757735]
  Large Language Models are emerging as powerful alternatives to traditional tree-based models.<n>Existing methods often rely on global linear proxies or scalar probability shifts that fail to capture the model's full probabilistic uncertainty.<n>We introduce TabSHAP, a model-agnostic interpretability framework designed to directly attribute local query decision logic.
  arXiv  Detail & Related papers  (2026-04-22T22:04:45Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• Scalable Causal Discovery from Recursive Nonlinear Data via Truncated Basis Function Scores and Tests [7.021824046220355]
  We introduce two basis-expansion tools for scalable causal discovery.<n>First, the Basis Function BIC score uses truncated additive expansions to approximate nonlinear dependencies.<n>Second, the Basis Function Likelihood Ratio Test (BF-LRT) provides an approximate conditional independence test.
  arXiv  Detail & Related papers  (2025-10-05T16:34:54Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysis [0.0]
  We present Datarus-R1-14B, a language model fine-tuned from Qwen 2.5-14B-Instruct to act as a virtual data analyst and graduate-level problem solver.<n>Datarus is trained not on isolated question-answer pairs but on full analytical trajectories including reasoning steps, code execution, error traces, self-corrections, and final conclusions.
  arXiv  Detail & Related papers  (2025-08-18T21:58:18Z)
• Unlocking Post-hoc Dataset Inference with Synthetic Data [11.886166976507711]
  Training datasets are often scraped from the internet without respecting data owners' intellectual property rights.<n>Inference (DI) offers a potential remedy by identifying whether a suspect dataset was used in training.<n>Existing DI methods require a private set-known to be absent from training-that closely matches the compromised dataset's distribution.<n>In this work, we address this challenge by synthetically generating the required held-out set.
  arXiv  Detail & Related papers  (2025-06-18T08:46:59Z)
• Effective Data Pruning through Score Extrapolation [40.61665742457229]
  We introduce a novel importance score extrapolation framework that requires training on only a small subset of data.<n>We present two initial approaches in this framework to accurately predict sample importance for the entire dataset using patterns learned from this minimal subset.<n>Our results indicate that score extrapolation is a promising direction to scale expensive score calculation methods, such as pruning, data attribution, or other tasks.
  arXiv  Detail & Related papers  (2025-06-10T17:38:49Z)
• When Simple Model Just Works: Is Network Traffic Classification in Crisis? [0.0]
  We show that a simple k-NN baseline using packet sequences metadata can be on par or even outperform more complex methods.<n>We argue that standard machine learning practices adapted from domains like NLP or computer vision may be ill-suited for network traffic classification.
  arXiv  Detail & Related papers  (2025-06-10T10:11:05Z)
• Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs [19.525112900768534]
  We show that models often appear to forget, but their original behavior can be rapidly restored with minimal fine-tuning.<n>We introduce a representation-level evaluation framework using PCA-based similarity and shift, centered kernel alignment, and Fisher information.<n>Applying this toolkit across six unlearning methods, three domains (text, code, math), and two open-source LLMs, we uncover a critical distinction between reversible and irreversible forgetting.
  arXiv  Detail & Related papers  (2025-05-22T16:02:10Z)
• Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness [53.96714099151378]
  We propose a three-step approach for parameter-efficient fine-tuning of image-text foundation models.<n>Our method improves its two key components: minority samples identification and the robust training algorithm.<n>Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error.
  arXiv  Detail & Related papers  (2025-03-12T15:46:12Z)
• Scaling Sparse and Dense Retrieval in Decoder-Only LLMs [20.173669986209024]
  Scaling large language models (LLMs) has shown great potential for improving retrieval model performance.<n>Previous studies have mainly focused on dense retrieval trained with contrastive loss (CL)<n>Sparse retrieval models consistently outperform dense retrieval across both in-domain (MSMARCO, TREC DL) and out-of-domain (BEIR) benchmarks.
  arXiv  Detail & Related papers  (2025-02-21T15:28:26Z)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal [70.28430200655919]
  We show that finetuning LLMs with verifier-based (VB) methods based on RL or search is far superior to verifier-free (VF) approaches based on distilling or cloning search traces, given a fixed amount of compute/data budget.<n>We corroborate our theory empirically on both didactic and math reasoning problems with 3/8B-sized pre-trained LLMs, where we find verification is crucial for scaling test-time compute.
  arXiv  Detail & Related papers  (2025-02-17T18:43:24Z)
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study [61.64685376882383]
  Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. This paper investigates the robustness of existing CLTR models in complex and diverse situations. We find that the DLA models and IPS-DCM show better robustness under various simulation settings than IPS-PBM and PRS with offline propensity estimation.
  arXiv  Detail & Related papers  (2024-04-04T10:54:38Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning [18.45898471459533]
  Spurious correlations that degrade model generalization or lead the model to be right for the wrong reasons are one of the main robustness concerns for real-world deployments. This paper proposes a novel approach to address spurious correlations during fine-tuning for a given domain of interest.
  arXiv  Detail & Related papers  (2023-04-08T05:20:33Z)
• Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
  We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV) NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones. We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
  arXiv  Detail & Related papers  (2022-06-23T09:35:03Z)
• Reliable validation of Reinforcement Learning Benchmarks [1.2031796234206134]
  Reinforcement Learning (RL) is one of the most dynamic research areas in Game AI and AI as a whole. There are numerous benchmark environments whose scores are used to compare different algorithms, such as Atari. We propose improving this situation by providing access to the original experimental data to validate study results.
  arXiv  Detail & Related papers  (2022-03-02T12:55:27Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• Training image classifiers using Semi-Weak Label Data [26.04162590798731]
  In Multiple Instance learning (MIL), weak labels are provided at the bag level with only presence/absence information known. This paper introduces a novel semi-weak label learning paradigm as a middle ground to mitigate the problem. We propose a two-stage framework to address the problem of learning from semi-weak labels.
  arXiv  Detail & Related papers  (2021-03-19T03:06:07Z)
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination [68.83098015578874]
  We integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination. CLD effectively brings unsupervised learning closer to natural data and real-world applications. New state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, and beats MoCo v2 and SimCLR on every reported performance.
  arXiv  Detail & Related papers  (2020-08-09T21:13:13Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.