Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise

• URL: http://arxiv.org/abs/2605.18022v1
• Date: Mon, 18 May 2026 08:12:45 GMT
• Title: Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise
• Abstract summary: We find that larger models tend to generalize better under appropriate optimization and model configurations, while noisy labels are memorized faster than clean data.<n>We propose a task-agnostic method to partition networks into generalization and memorization components.
• Score: 10.108345815812639
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Highly over-parameterized models can simultaneously memorize noisy labels and generalize well, yet how these behaviors coexist remains poorly understood. In this work, we investigate the underlying mechanisms of this coexistence using modular arithmetic tasks under heavy label noise. Through extensive experiments on two-layer neural networks, we find that larger models tend to generalize better under appropriate optimization and model configurations, while noisy labels are memorized faster than clean data. Over-parameterized models internally form a generalization structure, but its expression in the output is suppressed by the need to fit noisy labels. Remarkably, even with 80\% label noise, near-perfect test accuracy can be achieved by extracting this internal structure using frequency-based methods. We further propose a task-agnostic method to partition networks into generalization and memorization components. Although this subnetwork improves generalization, it is limited compared with frequency-based extraction, indicating that the generalization structure is distributed across neurons and motivating the development of new tools to retrieve generalizable knowledge from over-parameterized networks.

Related papers

• Beyond the Training Distribution: Mapping Generalization Boundaries in Neural Program Synthesis [13.460704771478866]
  Large-scale transformers achieve impressive results on program synthesis benchmarks.<n>But their true generalization capabilities remain obscured by data contamination and opaque training corpora.<n>We introduce a strictly controlled program synthesis environment based on a domain-specific arithmetic grammar.
  arXiv  Detail & Related papers  (2026-04-30T07:58:39Z)
• Variational Rectification Inference for Learning with Noisy Labels [74.85528327499662]
  We propose variational rectification inference (VRI) to formulate the adaptive rectification for loss functions.<n>VRI is constructed as a hierarchical Bayes by treating the rectifying vector as a latent variable.<n>By introducing the variational term in VRI, the conditional posterior is estimated accurately and avoids collapsing to a Dirac delta function.
  arXiv  Detail & Related papers  (2026-03-18T01:25:08Z)
• Generative Classifiers Avoid Shortcut Solutions [84.23247217037134]
  Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail under minor distribution shift.<n>We show that generative classifiers can avoid this issue by modeling all features, both core and spurious, instead of mainly spurious ones.<n>We find that diffusion-based and autorerimigressive generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• Feature Noise Boosts DNN Generalization under Label Noise [65.36889005555669]
  The presence of label noise in the training data has a profound impact on the generalization of deep neural networks (DNNs) In this study, we introduce and theoretically demonstrate a simple feature noise method, which directly adds noise to the features of training data.
  arXiv  Detail & Related papers  (2023-08-03T08:31:31Z)
• Fighting over-fitting with quantization for learning deep neural networks on noisy labels [7.09232719022402]
  We study the ability of compression methods to tackle both of these problems at once. We hypothesize that quantization-aware training, by restricting the expressivity of neural networks, behaves as a regularization.
  arXiv  Detail & Related papers  (2023-03-21T12:36:58Z)
• Predicting Out-of-Domain Generalization with Neighborhood Invariance [59.05399533508682]
  We propose a measure of a classifier's output invariance in a local transformation neighborhood. Our measure is simple to calculate, does not depend on the test point's true label, and can be applied even in out-of-domain (OOD) settings. In experiments on benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our measure and actual OOD generalization.
  arXiv  Detail & Related papers  (2022-07-05T14:55:16Z)
• Robust Training under Label Noise by Over-parameterization [41.03008228953627]
  We propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted. The main idea is yet very simple: label noise is sparse and incoherent with the network learned from clean data, so we model the noise and learn to separate it from the data. Remarkably, when trained using such a simple method in practice, we demonstrate state-of-the-art test accuracy against label noise on a variety of real datasets.
  arXiv  Detail & Related papers  (2022-02-28T18:50:10Z)
• Learning with Neighbor Consistency for Noisy Labels [69.83857578836769]
  We present a method for learning from noisy labels that leverages similarities between training examples in feature space. We evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, Clothing1M, mini-ImageNet-Red) noise.
  arXiv  Detail & Related papers  (2022-02-04T15:46:27Z)
• Information-Theoretic Generalization Bounds for Iterative Semi-Supervised Learning [81.1071978288003]
  In particular, we seek to understand the behaviour of the em generalization error of iterative SSL algorithms using information-theoretic principles. Our theoretical results suggest that when the class conditional variances are not too large, the upper bound on the generalization error decreases monotonically with the number of iterations, but quickly saturates.
  arXiv  Detail & Related papers  (2021-10-03T05:38:49Z)
• Learning from Noisy Labels for Entity-Centric Information Extraction [17.50856935207308]
  We propose a simple co-regularization framework for entity-centric information extraction. These models are jointly optimized with task-specific loss, and are regularized to generate similar predictions. In the end, we can take any of the trained models for inference.
  arXiv  Detail & Related papers  (2021-04-17T22:49:12Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)
• Generalization by Recognizing Confusion [3.018691733760647]
  Self-adaptive training technique augments modern neural networks by allowing them to adjust training labels on the fly. By combining the self-adaptive objective with mixup, we further improve the accuracy of self-adaptive models for image recognition. We find evidence that the Rademacher complexity of these algorithms is low, suggesting a new path towards provable generalization.
  arXiv  Detail & Related papers  (2020-06-13T22:49:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.