Deep Networks Favor Simple Data

• URL: http://arxiv.org/abs/2604.00394v2
• Date: Thu, 02 Apr 2026 01:01:54 GMT
• Title: Deep Networks Favor Simple Data
• Abstract summary: Estimated density is often interpreted as indicating how typical a sample is under a model.<n>Yet deep models trained on one dataset can assign higher density to simpler out-of-distribution (OOD) data than to in-distribution test data.<n>We separate the trained network from the density estimator built from its representations or outputs.
• Score: 10.927849392827406
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimated density is often interpreted as indicating how typical a sample is under a model. Yet deep models trained on one dataset can assign higher density to simpler out-of-distribution (OOD) data than to in-distribution test data. We refer to this behavior as the OOD anomaly. Prior work typically studies this phenomenon within a single architecture, detector, or benchmark, implicitly assuming certain canonical densities. We instead separate the trained network from the density estimator built from its representations or outputs. We introduce two estimators: Jacobian-based estimators and autoregressive self-estimators, making density analysis applicable to a wide range of models. Applying this perspective to a range of models, including iGPT, PixelCNN++, Glow, score-based diffusion models, DINOv2, and I-JEPA, we find the same striking regularity that goes beyond the OOD anomaly: lower-complexity samples receive higher estimated density, while higher-complexity samples receive lower estimated density. This ordering appears within a test set and across OOD pairs such as CIFAR-10 and SVHN, and remains highly consistent across independently trained models. To quantify these orderings, we introduce Spearman rank correlation and find striking agreement both across models and with external complexity metrics. Even when trained only on the lowest-density (most complex) samples - or even a single such sample - the resulting models still rank simpler images as higher density. These observations lead us beyond the original OOD anomaly to a more general conclusion: deep networks consistently favor simple data. Our goal is not to close this question, but to define and visualize it more clearly. We broaden its empirical scope and show that it appears across architectures, objectives, and density estimators.

Related papers

• Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine [60.669081685261965]
  Diffusion models generate high-dimensional data with remarkable quality.<n>How their training efficiently learns the score function remains theoretically unexplained.<n>We instantiate this principle as Score-induced Latent Diffusion (SiLD)
  arXiv  Detail & Related papers  (2026-05-16T16:51:10Z)
• On the Statistical Capacity of Deep Generative Models [10.288413514555861]
  We show that deep generative models can only generate concentrated samples that exhibit light tails.<n>These results shed light on the limited capacity of common deep generative models to handle heavy tails.
  arXiv  Detail & Related papers  (2025-01-14T00:39:46Z)
• Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think [53.2706196341054]
  We show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed.<n>We perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models.
  arXiv  Detail & Related papers  (2024-09-17T16:58:52Z)
• Out-of-Distribution Detection with a Single Unconditional Diffusion Model [54.15132801131365]
  Out-of-distribution (OOD) detection is a critical task in machine learning that seeks to identify abnormal samples. Traditionally, unsupervised methods utilize a deep generative model for OOD detection. This paper explores whether a single model can perform OOD detection across diverse tasks.
  arXiv  Detail & Related papers  (2024-05-20T08:54:03Z)
• Optimizing OOD Detection in Molecular Graphs: A Novel Approach with Diffusion Models [71.39421638547164]
  We propose to detect OOD molecules by adopting an auxiliary diffusion model-based framework, which compares similarities between input molecules and reconstructed graphs. Due to the generative bias towards reconstructing ID training samples, the similarity scores of OOD molecules will be much lower to facilitate detection. Our research pioneers an approach of Prototypical Graph Reconstruction for Molecular OOD Detection, dubbed as PGR-MOOD and hinges on three innovations.
  arXiv  Detail & Related papers  (2024-04-24T03:25:53Z)
• Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution Detection [2.3749120526936465]
  We show that deep generative models consistently infer higher log-likelihoods for OOD data than data they were trained on. We use the gradient of a data point with respect to the parameters of the deep generative model for OOD detection, based on the simple intuition that OOD data should have larger gradient norms than training data. Our empirical results indicate that this method outperforms the Typicality test for most deep generative models and image dataset pairings.
  arXiv  Detail & Related papers  (2024-03-03T11:36:35Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• Anomaly Detection with Variance Stabilized Density Estimation [49.46356430493534]
  We present a variance-stabilized density estimation problem for maximizing the likelihood of the observed samples. To obtain a reliable anomaly detector, we introduce a spectral ensemble of autoregressive models for learning the variance-stabilized distribution. We have conducted an extensive benchmark with 52 datasets, demonstrating that our method leads to state-of-the-art results.
  arXiv  Detail & Related papers  (2023-06-01T11:52:58Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Marginalizable Density Models [14.50261153230204]
  We present a novel deep network architecture which provides closed form expressions for the probabilities, marginals and conditionals of any subset of the variables. The model also allows for parallelized sampling with only a logarithmic dependence of the time complexity on the number of variables.
  arXiv  Detail & Related papers  (2021-06-08T23:54:48Z)
• Hierarchical VAEs Know What They Don't Know [6.649455007186671]
  We develop a fast, scalable and fully unsupervised likelihood-ratio score for OOD detection. We achieve state-of-the-art results on out-of-distribution detection.
  arXiv  Detail & Related papers  (2021-02-16T16:08:04Z)
• MOCCA: Multi-Layer One-Class ClassificAtion for Anomaly Detection [16.914663209964697]
  We propose our deep learning approach to the anomaly detection problem named Multi-LayerOne-Class Classification (MOCCA) We explicitly leverage the piece-wise nature of deep neural networks by exploiting information extracted at different depths to detect abnormal data instances. We show that our method reaches superior performances compared to the state-of-the-art approaches available in the literature.
  arXiv  Detail & Related papers  (2020-12-09T08:32:56Z)
• An Empirical Comparison of GANs and Normalizing Flows for Density Estimation [5.837881923712393]
  Generative adversarial networks (GANs) and normalizing flows are approaches to density estimation that use deep neural networks. GANs and normalizing flows have seldom been compared to each other for modeling non-image data. No GAN is capable of modeling our simple low-dimensional data well, a task we view as a prerequisite for an approach to be considered suitable for general-purpose statistical modeling.
  arXiv  Detail & Related papers  (2020-06-17T21:56:58Z)
• TraDE: Transformers for Density Estimation [101.20137732920718]
  TraDE is a self-attention-based architecture for auto-regressive density estimation. We present a suite of tasks such as regression using generated samples, out-of-distribution detection, and robustness to noise in the training data.
  arXiv  Detail & Related papers  (2020-04-06T07:32:51Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.