Adaptive Regularization for Large-Scale Sparse Feature Embedding Models

• URL: http://arxiv.org/abs/2511.06374v1
• Date: Sun, 09 Nov 2025 13:24:14 GMT
• Title: Adaptive Regularization for Large-Scale Sparse Feature Embedding Models
• Abstract summary: We provide a theoretical analysis that explains why overfitting occurs in models that use large-scale sparse categorical features.<n>Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch.
• Score: 1.0563630622294078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, they have not clearly identified the fundamental cause of this phenomenon. In this work, we provide a theoretical analysis that explains why overfitting occurs in models that use large-scale sparse categorical features. Based on this analysis, we propose an adaptive regularization method to address it. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems.

Related papers

• Does Normalization Choice Matter for Causal Large Time-Series Models? [3.974507021641083]
  We evaluate normalization strategies for transformer-based large time-series models trained with patching and efficient causal strategy.<n>We showcase that normalization choice significantly influences both training convergence and forecasting performance.
  arXiv  Detail & Related papers  (2026-06-08T10:51:56Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• Strong Linear Baselines Strike Back: Closed-Form Linear Models as Gaussian Process Conditional Density Estimators for TSAD [41.074068820031655]
  We show that a simple linear autoregressive anomaly score with the closed-form solution provided byOLS regression consistently matches or outperforms state-of-the-art deep detectors.<n>From a theoretical perspective, we show that linear models capture a broad class of anomaly types, estimating a finite-history Gaussian process conditional density.
  arXiv  Detail & Related papers  (2026-01-31T11:35:51Z)
• Did Models Sufficient Learn? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation [61.248535801314375]
  Subset-Selected Counterfactual Augmentation (SS-CA)<n>We develop Counterfactual LIMA to identify minimal spatial region sets whose removal can selectively alter model predictions.<n>Experiments show that SS-CA improves generalization on in-distribution (ID) test data and achieves superior performance on out-of-distribution (OOD) benchmarks.
  arXiv  Detail & Related papers  (2025-11-15T08:39:22Z)
• Characteristic Root Analysis and Regularization for Linear Time Series Forecasting [9.254995889539716]
  Time series forecasting remains a critical challenge across numerous domains.<n>Recent studies highlight the surprising competitiveness of simple linear models.<n>This paper focuses on the role of characteristic roots in temporal dynamics.
  arXiv  Detail & Related papers  (2025-09-28T03:06:30Z)
• Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models [57.49136894315871]
  New paradigm of test-time scaling has yielded remarkable breakthroughs in reasoning models and generative vision models.<n>We propose one solution to the problem of integrating test-time scaling knowledge into a model during post-training.<n>We replace reward guided test-time noise optimization in diffusion models with a Noise Hypernetwork that modulates initial input noise.
  arXiv  Detail & Related papers  (2025-08-13T17:33:37Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Salvaging the Overlooked: Leveraging Class-Aware Contrastive Learning for Multi-Class Anomaly Detection [18.797864512898787]
  In anomaly detection, early approaches often train separate models for individual classes, yielding high performance but posing challenges in scalability and resource management.<n>We investigate this performance observed in reconstruction-based methods, identifying the key issue: inter-class confusion.<n>This confusion emerges when a model trained in multi-class scenarios incorrectly reconstructs samples from one class as those of another, thereby exacerbating reconstruction errors.<n>By explicitly leveraging raw object category information (eg carpet or wood), we introduce local CL to refine multiscale dense features, and global CL to obtain more compact feature representations of normal patterns, thereby effectively adapting the models to multi-class
  arXiv  Detail & Related papers  (2024-12-06T04:31:09Z)
• ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization [0.32885740436059047]
  Over- parameterized neural network models often lead to significant performance discrepancies between training and test sets.<n>We introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets.<n>We propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set.
  arXiv  Detail & Related papers  (2024-12-02T13:21:31Z)
• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms [50.808123629394245]
  Direct Alignment Algorithms (DDAs) like Direct Preference Optimization have emerged as alternatives to the classical RLHF pipeline. This work formulates and formalizes the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.
  arXiv  Detail & Related papers  (2024-06-05T03:41:37Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds [44.79535333220044]
  Recent studies have shown that tail classes are not always hard to learn, and model bias has been observed on sample-balanced datasets.<n>In this work, we first establish a geometric perspective for analyzing model fairness and then systematically propose a series of geometric measurements.
  arXiv  Detail & Related papers  (2023-03-22T04:49:23Z)
• Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models [16.984947259260878]
  We observe an interesting one-epoch overfitting problem in Click-Through Rate (CTR) prediction. The model performance exhibits a dramatic degradation at the beginning of the second epoch. Thereby, the best performance is usually achieved by training with only one epoch.
  arXiv  Detail & Related papers  (2022-09-04T11:36:16Z)
• Consistent Counterfactuals for Deep Models [25.1271020453651]
  Counterfactual examples are used to explain predictions of machine learning models in key areas such as finance and medical diagnosis. This paper studies the consistency of model prediction on counterfactual examples in deep networks under small changes to initial training conditions.
  arXiv  Detail & Related papers  (2021-10-06T23:48:55Z)
• On Model Calibration for Long-Tailed Object Detection and Instance Segmentation [56.82077636126353]
  We propose NorCal, Normalized for long-tailed object detection and instance segmentation. We show that separately handling the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance.
  arXiv  Detail & Related papers  (2021-07-05T17:57:20Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.