Mitigating Early Training Collapse in CTR Models

• URL: http://arxiv.org/abs/2607.09696v1
• Date: Sat, 20 Jun 2026 04:34:39 GMT
• Title: Mitigating Early Training Collapse in CTR Models
• Abstract summary: Deep neural models for click-through rate prediction often exhibit a sharp decline in validation performance immediately after the first training epoch.<n>In this study, we analyze this behavior using large-scale industrial datasets and evaluate practical mitigation strategies.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural models for click-through rate prediction often exhibit a sharp decline in validation performance immediately after the first training epoch despite continued improvement in training loss. This instability restricts effective learning and limits model performance. In this study, we analyze this behavior using large-scale industrial datasets and evaluate practical mitigation strategies. While reducing the learning rate provides only incremental gains, controlling feature sparsity yields substantial improvements. Removing highly sparse features and aggregating infrequent feature values stabilizes training, extends useful learning beyond a single epoch, and improves both offline evaluation metrics and online system performance.

Related papers

• Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates [10.06646055401747]
  Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining.<n>Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences.<n>We introduce FINCH, a loss-adaptive learning-rate schedule that reduces forgetting by 93% on average while matching the task performance of standard fine-tuning.
  arXiv  Detail & Related papers  (2026-05-19T15:36:52Z)
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data [89.96277093034547]
  We introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization.<n>We show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training.
  arXiv  Detail & Related papers  (2025-12-29T12:35:51Z)
• Knowledge-guided Continual Learning for Behavioral Analytics Systems [0.3437656066916039]
  Models that learn to capture this content can experience a decrease in performance over time due to data drift.<n>Fine-tuning such a model over time with new data can be detrimental due to catastrophic forgetting.<n>Replay-based approaches in continual learning offer a simple yet efficient method to update such models.<n>We propose a novel augmentation-based approach to incorporate external knowledge in the replay-based continual learning framework.
  arXiv  Detail & Related papers  (2025-10-25T19:04:14Z)
• Deep Generative Continual Learning using Functional LoRA: FunLoRA [12.547444644243543]
  A common strategy consists in retraining the generative model on its own synthetic data in order to mitigate forgetting.<n>We propose a novel and more expressive conditioning mechanism for generative models based on low rank adaptation (LoRA)<n>Our proposed parameter-efficient fine-tuning (PEFT) method surpasses prior state-of-the-art results based on diffusion models.
  arXiv  Detail & Related papers  (2025-10-03T00:18:05Z)
• Instance-dependent Early Stopping [57.912273923450726]
  We propose an Instance-dependent Early Stopping (IES) method that adapts the early stopping mechanism from the entire training set to the instance level.<n>IES considers an instance as mastered if the second-order differences of its loss value remain within a small range around zero.<n>IES can reduce backpropagation instances by 10%-50% while maintaining or even slightly improving the test accuracy and transfer learning performance of a model.
  arXiv  Detail & Related papers  (2025-02-11T13:34:09Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• Churn Reduction via Distillation [54.5952282395487]
  We show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines.
  arXiv  Detail & Related papers  (2021-06-04T18:03:31Z)
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning [97.28695683236981]
  More gradient updates decrease the expressivity of the current value network. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings.
  arXiv  Detail & Related papers  (2020-10-27T17:55:16Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Overfitting in adversarially robust deep learning [86.11788847990783]
  We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
  arXiv  Detail & Related papers  (2020-02-26T15:40:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.