Efficient Cross-Architecture Knowledge Transfer for Large-Scale Online User Response Prediction

• URL: http://arxiv.org/abs/2602.01775v1
• Date: Mon, 02 Feb 2026 08:00:36 GMT
• Title: Efficient Cross-Architecture Knowledge Transfer for Large-Scale Online User Response Prediction
• Abstract summary: CrossAdapt is a two-stage framework for efficient cross-architecture knowledge transfer.<n>The offline stage enables rapid embedding transfer via dimension-adaptive projections without iterative training.<n>The online stage introduces asymmetric co-distillation, where students update frequently while teachers update infrequently.
• Score: 43.353811451921466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying new architectures in large-scale user response prediction systems incurs high model switching costs due to expensive retraining on massive historical data and performance degradation under data retention constraints. Existing knowledge distillation methods struggle with architectural heterogeneity and the prohibitive cost of transferring large embedding tables. We propose CrossAdapt, a two-stage framework for efficient cross-architecture knowledge transfer. The offline stage enables rapid embedding transfer via dimension-adaptive projections without iterative training, combined with progressive network distillation and strategic sampling to reduce computational cost. The online stage introduces asymmetric co-distillation, where students update frequently while teachers update infrequently, together with a distribution-aware adaptation mechanism that dynamically balances historical knowledge preservation and fast adaptation to evolving data. Experiments on three public datasets show that CrossAdapt achieves 0.27-0.43% AUC improvements while reducing training time by 43-71%. Large-scale deployment on Tencent WeChat Channels (~10M daily samples) further demonstrates its effectiveness, significantly mitigating AUC degradation, LogLoss increase, and prediction bias compared to standard distillation baselines.

Related papers

• Multimodal Dataset Distillation via Phased Teacher Models [46.18186490102602]
  Phased Teacher Model with Shortcut Trajectory (PTM-ST) is a novel phased distillation framework.<n>We show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps.<n>Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO.
  arXiv  Detail & Related papers  (2026-03-26T12:38:12Z)
• Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward [54.708851958671794]
  We propose a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection.<n>In offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty.<n>During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential.
  arXiv  Detail & Related papers  (2025-09-01T10:04:20Z)
• Light distillation for Incremental Graph Convolution Collaborative Filtering [0.0]
  We propose a light preference-driven distillation method to distill the preference score of a user for an item directly from historical interactions.<n>The proposed method can save training time from 1.5x to 9.5x compared to the existing methods and improves Recall@20 by 5.41% and 10.64% from the fine-tune method.
  arXiv  Detail & Related papers  (2025-05-26T10:47:26Z)
• SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning [51.10866035483686]
  High update-to-data (UTD) ratio algorithms in reinforcement learning (RL) improve sample efficiency but incur high computational costs, limiting real-world scalability.<n>We propose Offline Stabilization Phases for Efficient Q-Learning (SPEQ), an RL algorithm that combines low-UTD online training with periodic offline stabilization phases.<n>During these phases, Q-functions are fine-tuned with high UTD ratios on a fixed replay buffer, reducing redundant updates on suboptimal data.
  arXiv  Detail & Related papers  (2025-01-15T09:04:19Z)
• Efficient and Effective Augmentation Strategy for Adversarial Training [48.735220353660324]
  Adversarial training of Deep Neural Networks is known to be significantly more data-hungry than standard training. We propose Diverse Augmentation-based Joint Adversarial Training (DAJAT) to use data augmentations effectively in adversarial training.
  arXiv  Detail & Related papers  (2022-10-27T10:59:55Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability [53.27240222619834]
  Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks. Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
  arXiv  Detail & Related papers  (2022-03-10T06:23:41Z)
• Accumulated Polar Feature-based Deep Learning for Efficient and Lightweight Automatic Modulation Classification with Channel Compensation Mechanism [6.915743897443897]
  In next-generation communications, massive machine-type communications (mMTC) induce severe burden on base stations. Deep learning (DL) technique stores intelligence in the network, resulting in superior performance over traditional approaches. In this work, an accumulated polar feature-based DL with a channel compensation mechanism is proposed to cope with the aforementioned issues.
  arXiv  Detail & Related papers  (2020-01-06T04:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.