Random Label Prediction Heads for Studying Memorization in Deep Neural Networks

• URL: http://arxiv.org/abs/2607.11541v1
• Date: Mon, 13 Jul 2026 13:26:42 GMT
• Title: Random Label Prediction Heads for Studying Memorization in Deep Neural Networks
• Abstract summary: We introduce a straightforward yet effective method to empirically study memorization in deep neural networks for classification tasks.<n>Our approach augments each training sample with auxiliary random labels, which are then predicted by a random label prediction head (RLP-head)<n>We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets.
• Score: 3.134883909555613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a straightforward yet effective method to empirically study memorization in deep neural networks for classification tasks. Our approach augments each training sample with auxiliary random labels, which are then predicted by a random label prediction head (RLP-head). RLP-heads can be attached at arbitrary depths of a network, predicting random labels from the corresponding intermediate representation and thereby enabling analysis of how memorization capacity evolves across layers. By interpreting the RLP-head performance as an empirical estimate of Rademacher complexity, we obtain a direct measure of both sample-level memorization and model capacity. We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets. Building on this approach, we further propose a novel regularization technique based on the output of the RLP-head, which demonstrably reduces memorization. Interestingly, our experiments reveal that reducing memorization can either improve or impair generalization, depending on the dataset and training setup. These findings challenge the traditional assumption that overfitting is equivalent to memorization and suggest new hypotheses to reconcile these seemingly contradictory results. The source code is available at https://github.com/MarlonBecker/RandomLabelHeads

Related papers

• Amortized Variational Inference for Partial-Label Learning: A Probabilistic Approach to Label Disambiguation [2.7214777196418645]
  Partial-label learning trains classifiers when each instance is associated with a set of candidate labels, only one of which is correct.<n>We introduce a novel framework that directly approximates the posterior distribution over true labels using amortized variational inference.<n>Our method employs neural networks to predict variational parameters from input data, enabling efficient inference.
  arXiv  Detail & Related papers  (2025-10-24T09:54:23Z)
• Simultaneous Automatic Picking and Manual Picking Refinement for First-Break [9.869865340964576]
  We introduce the Simultaneous Picking and Refinement (SPR) algorithm, designed to handle datasets plagued by outliers or even noisy labels.<n>Unlike conventional approaches that regard manual picks as ground truth, our method treats the true first-break as a latent variable within a probabilistic model.<n>SPR provides a robust solution to some of the principal obstacles in automatic first-break picking.
  arXiv  Detail & Related papers  (2025-02-03T16:05:57Z)
• Learned Random Label Predictions as a Neural Network Complexity Metric [0.6577148087211809]
  We investigate the impact of learning randomly generated labels in parallel to class labels in supervised learning on memorization, model complexity, and generalization in deep neural networks.<n>Inspired by methods used in fair AI, our approach allows for the unlearning of random labels, preventing the network from memorizing individual samples.
  arXiv  Detail & Related papers  (2024-11-29T11:52:59Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification [49.09505771145326]
  We propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels. Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-04-26T06:00:27Z)
• LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification [28.37907856670151]
  Pseudo-labels are noisy due to their nature, so selecting the correct ones has a huge potential for performance boost. We propose a novel pseudo-label selection method LOPS that memorize takes learning order of samples into consideration. LOPS can be viewed as a strong performance-boost plug-in to most of existing weakly-supervised text classification methods.
  arXiv  Detail & Related papers  (2022-05-25T06:46:48Z)
• Transformers Can Do Bayesian Inference [56.99390658880008]
  We present Prior-Data Fitted Networks (PFNs) PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems.
  arXiv  Detail & Related papers  (2021-12-20T13:07:39Z)
• Exploring Memorization in Adversarial Training [58.38336773082818]
  We investigate the memorization effect in adversarial training (AT) for promoting a deeper understanding of capacity, convergence, generalization, and especially robust overfitting. We propose a new mitigation algorithm motivated by detailed memorization analyses.
  arXiv  Detail & Related papers  (2021-06-03T05:39:57Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.