DIET: Learning to Distill Dataset Continually for Recommender Systems

• URL: http://arxiv.org/abs/2603.24958v1
• Date: Thu, 26 Mar 2026 02:50:13 GMT
• Title: DIET: Learning to Distill Dataset Continually for Recommender Systems
• Abstract summary: Deep recommender models are trained under a continual learning paradigm, relying on massive and continuously growing streaming behavioral logs.<n>In large-scale platforms, retraining models on full historical data for architecture comparison or iteration is prohibitively expensive, severely slowing down model development.<n>We formulate this problem as emphstreaming dataset distillation for recommender systems and propose textbfDIET, a unified framework that maintains a compact distilled dataset which evolves alongside streaming data while preserving training-critical signals.
• Score: 69.68271683922536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern deep recommender models are trained under a continual learning paradigm, relying on massive and continuously growing streaming behavioral logs. In large-scale platforms, retraining models on full historical data for architecture comparison or iteration is prohibitively expensive, severely slowing down model development. This challenge calls for data-efficient approaches that can faithfully approximate full-data training behavior without repeatedly processing the entire evolving data stream. We formulate this problem as \emph{streaming dataset distillation for recommender systems} and propose \textbf{DIET}, a unified framework that maintains a compact distilled dataset which evolves alongside streaming data while preserving training-critical signals. Unlike existing dataset distillation methods that construct a static distilled set, DIET models distilled data as an evolving training memory and updates it in a stage-wise manner to remain aligned with long-term training dynamics. DIET enables effective continual distillation through principled initialization from influential samples and selective updates guided by influence-aware memory addressing within a bi-level optimization framework. Experiments on large-scale recommendation benchmarks demonstrate that DIET compresses training data to as little as \textbf{1-2\%} of the original size while preserving performance trends consistent with full-data training, reducing model iteration cost by up to \textbf{60$\times$}. Moreover, the distilled datasets produced by DIET generalize well across different model architectures, highlighting streaming dataset distillation as a scalable and reusable data foundation for recommender system development.

Related papers

• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods [30.96121293915128]
  ADAPT is a dynamic online framework that reweights training samples with adaptive per-sample learning rates guided by similarity-based quality signals.<n> ADAPT consistently outperforms offline selection/mixing and prior online methods, achieving stronger cross-benchmark generalization under equal FLOPs.
  arXiv  Detail & Related papers  (2026-04-19T14:23:23Z)
• Learnability-Guided Diffusion for Dataset Distillation [51.56484100374058]
  We propose learnability-driven dataset distillation, which constructs synthetic datasets incrementally through successive stages.<n>We introduce Learnability-Guided Diffusion (LGD), which balances training utility for the current model with validity under a reference model to generate curriculum-aligned samples.<n>Our approach reduces redundancy by 39.1%, promotes specialization across training stages, and achieves state-of-the-art results on ImageNet-1K (60.1%), ImageNette (87.2%), and ImageWoof (72.9%)
  arXiv  Detail & Related papers  (2026-04-01T06:04:59Z)
• DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models [51.48564522455171]
  We present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory.<n>DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment adjustment, and data reweighting.<n>It provides trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training.
  arXiv  Detail & Related papers  (2026-03-27T08:28:02Z)
• Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice [109.9635246405237]
  We show that the experiment conclusions about data quality can flip with even minor adjustments to training hyper parameters.<n>We introduce a simple patch to the evaluation protocol: using reduced learning rates for proxy model training.<n> Empirically, we validate this approach across 23 data recipes covering four critical dimensions of data curation.
  arXiv  Detail & Related papers  (2025-12-30T23:02:44Z)
• Restoration Score Distillation: From Corrupted Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We propose textitRestoration Score Distillation (RSD), a principled generalization of Denoising Score Distillation (DSD)<n>RSD accommodates a broader range of corruption types, such as blurred, incomplete, or low-resolution images.<n>It consistently surpasses its teacher model across diverse restoration tasks on both natural and scientific datasets.
  arXiv  Detail & Related papers  (2025-05-19T17:21:03Z)
• SCAN: Bootstrapping Contrastive Pre-training for Data Efficiency [10.555957282859]
  This paper introduces a novel dynamic bootstrapping dataset pruning method. It involves pruning data preparation followed by dataset mutation operations, both of which undergo iterative and dynamic updates. We individually pre-train seven CLIP models on two large-scale image-text pair datasets, and two MoCo models on the ImageNet dataset, resulting in a total of 16 pre-trained models.
  arXiv  Detail & Related papers  (2024-11-14T01:53:17Z)
• Self-supervised Dataset Distillation: A Good Compression Is All You Need [23.02066055996762]
  We introduce SC-DD, a simple yet effective Self-supervised Compression framework for dataset distillation. The proposed SC-DD outperforms all previous state-of-the-art supervised dataset distillation methods when employing larger models. Experiments are conducted on CIFAR-100, Tiny-ImageNet and ImageNet-1K datasets to demonstrate the superiority of our proposed approach.
  arXiv  Detail & Related papers  (2024-04-11T17:56:40Z)
• Towards Adversarially Robust Dataset Distillation by Curvature Regularization [11.02948004359488]
  dataset distillation (DD) allows datasets to be distilled to fractions of their original size while preserving the rich distributional information.<n>Recent research in this area has been focusing on improving the accuracy of models trained on distilled datasets.<n>We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training.
  arXiv  Detail & Related papers  (2024-03-15T06:31:03Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• Dataset Distillation: A Comprehensive Review [76.26276286545284]
  dataset distillation (DD) aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. This paper gives a comprehensive review and summary of recent advances in DD and its application.
  arXiv  Detail & Related papers  (2023-01-17T17:03:28Z)
• Accelerating Dataset Distillation via Model Augmentation [41.3027484667024]
  We propose two model augmentation techniques, i.e. using early-stage models and parameter parameters to learn an informative synthetic set with significantly reduced training cost. Our method achieves up to 20x speedup and comparable performance on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-12-12T07:36:05Z)
• Dataset Distillation using Neural Feature Regression [32.53291298089172]
  We develop an algorithm for dataset distillation using neural Feature Regression with Pooling (FRePo) FRePo achieves state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. We show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense.
  arXiv  Detail & Related papers  (2022-06-01T19:02:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.