What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

• URL: http://arxiv.org/abs/2606.29791v1
• Date: Mon, 29 Jun 2026 05:09:20 GMT
• Title: What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
• Abstract summary: Outlier detection aims to identify anomalous instances by learning the structure of normal data (inliers)<n>Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers.<n>We show that a simple autoencoder can successfully memorize inliers while failing to memorize outliers during certain stages of early training.
• Score: 7.78867825272241
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.

Related papers

• Imperfect Influence, Preserved Rankings: A Theory of TRAK for Data Attribution [10.662938465357158]
  We provide a theoretical analysis of the TRAK algorithm, characterizing its performance and quantifying the errors introduced by the approximations on which the method relies.<n>We show that although the approximations incur significant errors, TRAK's estimated influence remains highly correlated with the original influence and therefore largely preserves the relative ranking of data points.
  arXiv  Detail & Related papers  (2026-02-01T16:11:49Z)
• Memorize Early, Then Query: Inlier-Memorization-Guided Active Outlier Detection [5.34972604411447]
  Inlier-memorization (IM) effect provides a promising signal for distinguishing outliers.<n>Existing unsupervised approaches that rely solely on the IM effect still struggle when inliers and outliers are not well-separated.<n>We propose IMBoost, a novel framework that explicitly reinforces the IM effect to improve outlier detection.
  arXiv  Detail & Related papers  (2026-01-16T04:55:46Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• ALTBI: Constructing Improved Outlier Detection Models via Optimization of Inlier-Memorization Effect [2.3961612657966946]
  Outlier detection (OD) is the task of identifying unusual observations (or outliers) from a given or upcoming data.<n>Inlier-memorization (IM) effect suggests that generative models memorize inliers before outliers in early learning stages.<n>We propose a theoretically principled method to address UOD tasks by maximally utilizing the IM effect.
  arXiv  Detail & Related papers  (2024-08-19T08:40:53Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• PREM: A Simple Yet Effective Approach for Node-Level Graph Anomaly Detection [65.24854366973794]
  Node-level graph anomaly detection (GAD) plays a critical role in identifying anomalous nodes from graph-structured data in domains such as medicine, social networks, and e-commerce. We introduce a simple method termed PREprocessing and Matching (PREM for short) to improve the efficiency of GAD. Our approach streamlines GAD, reducing time and memory consumption while maintaining powerful anomaly detection capabilities.
  arXiv  Detail & Related papers  (2023-10-18T02:59:57Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• ODIM: Outlier Detection via Likelihood of Under-Fitted Generative Models [4.956259629094216]
  unsupervised outlier detection (UOD) problem refers to a task to identify inliers given training data which contain outliers as well as inliers. We develop a new method called the outlier detection via the IM effect (ODIM) Remarkably, the ODIM requires only a few updates, making it computationally efficient at least tens of times faster than other deep-learning-based algorithms.
  arXiv  Detail & Related papers  (2023-01-11T01:02:27Z)
• Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees [1.2289361708127877]
  Marginal contribution feature importance (MCI) was developed to quantifying the relationships in data. We introduce ultra-marginal feature importance (UMFI), which uses dependence removal techniques from the AI fairness literature as its foundation. We show on real and simulated data that UMFI performs better than MCI, especially in the presence of correlated interactions and unrelated features.
  arXiv  Detail & Related papers  (2022-04-21T07:54:58Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.