Stable but Wrong: When More Data Degrades Scientific Conclusions

• URL: http://arxiv.org/abs/2602.05668v1
• Date: Thu, 05 Feb 2026 13:51:47 GMT
• Title: Stable but Wrong: When More Data Degrades Scientific Conclusions
• Abstract summary: We show that modern science increasingly relies on ever-growing observational datasets and automated inference pipelines.<n>We identify a structural regime in which standard inference procedures converge smoothly, remain well calibrated, and pass conventional diagnostic checks.<n>This failure arises when the reliability of observations degrades in a manner that is intrinsically unobservable to the inference process itself.
• Score: 26.934158434915133
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern science increasingly relies on ever-growing observational datasets and automated inference pipelines, under the implicit belief that accumulating more data makes scientific conclusions more reliable. Here we show that this belief can fail in a fundamental and irreversible way. We identify a structural regime in which standard inference procedures converge smoothly, remain well calibrated, and pass conventional diagnostic checks, yet systematically converge to incorrect conclusions. This failure arises when the reliability of observations degrades in a manner that is intrinsically unobservable to the inference process itself. Using minimal synthetic experiments, we demonstrate that in this regime additional data do not correct error but instead amplify it, while residual-based and goodness-of-fit diagnostics remain misleadingly normal. These results reveal an intrinsic limit of data-driven science: stability, convergence, and confidence are not sufficient indicators of epistemic validity. We argue that inference cannot be treated as an unconditional consequence of data availability, but must instead be governed by explicit constraints on the integrity of the observational process.

Related papers

• Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components [2.439525434311725]
  In domains relying on emphindirect observation, the learner observes a derivative representation produced by multi-stage measurement, reconstruction, and preprocessing pipelines.<n>We identify three failure modes arising from this frozen lens''<n>We call on the AI4Science community to make pipelines emphcomputable inference objects via domain-specific Computable Observation Frameworks.
  arXiv  Detail & Related papers  (2026-05-23T12:50:00Z)
• The Digital Twin Counterfactual Framework: A Validation Architecture for Simulated Potential Outcomes [0.0]
  Digital Twin Counterfactual Framework (DTCF)<n>Five-level validation architecture converts unfalsifiable claim that simulator produces correct counterfactuals into falsifiable tests against observable data.<n> decomposition separates causal quantities into those that are marginally validated.<n>Joint causal claims become explicitly assumption-indexed.
  arXiv  Detail & Related papers  (2026-04-01T19:04:15Z)
• Effect-Level Validation for Causal Discovery [1.8192444294441061]
  Causal discovery is increasingly applied to large-scale telemetry data to estimate the effects of user-facing interventions.<n>But its reliability for decision-making in feedback-driven systems with strong self-selection remains unclear.<n>We propose an effect-centric, admissibility-first framework that treats discovered graphs as structural hypotheses.
  arXiv  Detail & Related papers  (2026-02-09T07:26:55Z)
• "I May Not Have Articulated Myself Clearly": Diagnosing Dynamic Instability in LLM Reasoning at Inference Time [1.570719611178015]
  Reasoning failures in large language models (LLMs) are typically measured only at the end of a generation, yet many failures manifest as a process-level breakdown.<n>We define a simple instability signal that combines consecutive-step distributional shift (JSD) and uncertainty (entropy)<n>Across GSM8K and HotpotQA, instability strength predicts wrong answers with above-chance AUC and yields monotonic bucket-level accuracy decline.
  arXiv  Detail & Related papers  (2026-02-02T22:11:25Z)
• Cross-Validated Causal Inference: a Modern Method to Combine Experimental and Observational Data [48.72384067821617]
  We develop new methods to integrate experimental and observational data in causal inference.<n>A full model containing the causal parameter is obtained by minimizing a weighted combination of experimental and observational losses.<n>Experiments on real and synthetic data show the efficacy and reliability of our method.
  arXiv  Detail & Related papers  (2025-11-01T22:24:16Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• The Hardness of Validating Observational Studies with Experimental Data [2.9593087583214173]
  We show that it is possible to use experimental data to emphfalsify causal effect estimates from observational data.<n>Our theorem proves that while experimental data can be used to detect bias in observational studies, without additional assumptions on the smoothness of the correction function, it can not be used to remove it.
  arXiv  Detail & Related papers  (2025-03-19T00:06:23Z)
• Doubly robust identification of treatment effects from multiple environments [22.228179404621482]
  We propose RAMEN, an algorithm that produces unbiased treatment effect estimates without the need to know or learn the underlying causal graph.<n> RAMEN achieves doubly robust identification: it can identify the treatment effect whenever the causal parents of the treatment or those of the outcome are observed.
  arXiv  Detail & Related papers  (2025-03-18T17:33:10Z)
• DAGnosis: Localized Identification of Data Inconsistencies using Structures [73.39285449012255]
  Identification and appropriate handling of inconsistencies in data at deployment time is crucial to reliably use machine learning models. We use directed acyclic graphs (DAGs) to encode the training set's features probability distribution and independencies as a structure. Our method, called DAGnosis, leverages these structural interactions to bring valuable and insightful data-centric conclusions.
  arXiv  Detail & Related papers  (2024-02-26T11:29:16Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Causal Discovery via Conditional Independence Testing with Proxy Variables [35.3493980628004]
  The presence of unobserved variables, such as the latent confounder, can introduce bias in conditional independence testing. We propose a novel hypothesis-testing procedure that can effectively examine the existence of the causal relationship over continuous variables.
  arXiv  Detail & Related papers  (2023-05-09T09:08:39Z)
• BaCaDI: Bayesian Causal Discovery with Unknown Interventions [118.93754590721173]
  BaCaDI operates in the continuous space of latent probabilistic representations of both causal structures and interventions. In experiments on synthetic causal discovery tasks and simulated gene-expression data, BaCaDI outperforms related methods in identifying causal structures and intervention targets.
  arXiv  Detail & Related papers  (2022-06-03T16:25:48Z)
• Causal Discovery from Conditionally Stationary Time Series [14.297325665581353]
  We develop a causal discovery approach to handle a wide class of nonstationary time series.<n>Named State-Dependent Causal Inference (SDCI), our approach is able to recover the underlying causal dependencies.<n> Empirical experiments on nonlinear particle interaction data and gene regulatory networks demonstrate SDCI's superior performance.
  arXiv  Detail & Related papers  (2021-10-12T18:12:57Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.