Forward Once for All: Structural Parameterized Adaptation for Efficient Cloud-coordinated On-device Recommendation

• URL: http://arxiv.org/abs/2501.02837v1
• Date: Mon, 06 Jan 2025 08:32:16 GMT
• Title: Forward Once for All: Structural Parameterized Adaptation for Efficient Cloud-coordinated On-device Recommendation
• Authors: Kairui Fu, Zheqi Lv, Shengyu Zhang, Fan Wu, Kun Kuang,
• Abstract summary: Forward-OFA is a novel approach for the dynamic construction of device-specific networks. It establishes a structure-guided mapping of real-time behaviors to the parameters of assembled networks. Experiments on real-world datasets demonstrate the effectiveness and efficiency of Forward-OFA.
• Score: 26.353286155116116
• License:
• Abstract: In cloud-centric recommender system, regular data exchanges between user devices and cloud could potentially elevate bandwidth demands and privacy risks. On-device recommendation emerges as a viable solution by performing reranking locally to alleviate these concerns. Existing methods primarily focus on developing local adaptive parameters, while potentially neglecting the critical role of tailor-made model architecture. Insights from broader research domains suggest that varying data distributions might favor distinct architectures for better fitting. In addition, imposing a uniform model structure across heterogeneous devices may result in risking inefficacy on less capable devices or sub-optimal performance on those with sufficient capabilities. In response to these gaps, our paper introduces Forward-OFA, a novel approach for the dynamic construction of device-specific networks (both structure and parameters). Forward-OFA employs a structure controller to selectively determine whether each block needs to be assembled for a given device. However, during the training of the structure controller, these assembled heterogeneous structures are jointly optimized, where the co-adaption among blocks might encounter gradient conflicts. To mitigate this, Forward-OFA is designed to establish a structure-guided mapping of real-time behaviors to the parameters of assembled networks. Structure-related parameters and parallel components within the mapper prevent each part from receiving heterogeneous gradients from others, thus bypassing the gradient conflicts for coupled optimization. Besides, direct mapping enables Forward-OFA to achieve adaptation through only one forward pass, allowing for swift adaptation to changing interests and eliminating the requirement for on-device backpropagation. Experiments on real-world datasets demonstrate the effectiveness and efficiency of Forward-OFA.

Related papers

• Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edge [35.40849522296486]
  Large-scale foundation models (FoMos) can perform human-like intelligence. FoMos need to be adapted to specialized downstream tasks through fine-tuning techniques. We advocate multi-device cooperation within the device-edge cooperative fine-tuning paradigm.
  arXiv  Detail & Related papers  (2024-07-13T12:47:14Z)
• Enhancing Federated Learning with Adaptive Differential Privacy and Priority-Based Aggregation [0.0]
  Federated learning (FL) develops global models without direct access to local datasets. It is possible to access the model updates transferred between clients and servers, potentially revealing sensitive local information to adversaries. Differential privacy (DP) offers a promising approach to addressing this issue by adding noise to the parameters. We propose a personalized DP framework that injects noise based on clients' relative impact factors and aggregates parameters.
  arXiv  Detail & Related papers  (2024-06-26T16:55:07Z)
• Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model [81.55141188169621]
  We equip PEFT with a cross-block orchestration mechanism to enable the adaptation of the Segment Anything Model (SAM) to various downstream scenarios. We propose an intra-block enhancement module, which introduces a linear projection head whose weights are generated from a hyper-complex layer. Our proposed approach consistently improves the segmentation performance significantly on novel scenarios with only around 1K additional parameters.
  arXiv  Detail & Related papers  (2023-11-28T11:23:34Z)
• Energizing Federated Learning via Filter-Aware Attention [39.17451229130728]
  Federated learning (FL) is a promising distributed paradigm, eliminating the need for data sharing but facing challenges from data heterogeneity. We propose FedOFA, utilizing personalized filter attention for parameter recalibration. The core is the Two-stream Filter-aware Attention (TFA) module, meticulously designed to extract personalized filter-aware attention maps. AGPS selectively retains crucial neurons while masking redundant ones, reducing communication costs without performance sacrifice.
  arXiv  Detail & Related papers  (2023-11-18T09:09:38Z)
• Subspace-Configurable Networks [16.786433652213013]
  Deep learning models on edge devices often lack robustness when faced with dynamic changes in sensed data. In this paper, we train a parameterized subspace of networks, where an optimal network for a particular parameter setting is part of this subspace. The obtained subspace is low-dimensional and has a surprisingly simple structure even for complex, non-invertible transformations of the input.
  arXiv  Detail & Related papers  (2023-05-22T23:19:45Z)
• Adaptive Spot-Guided Transformer for Consistent Local Feature Matching [64.30749838423922]
  We propose Adaptive Spot-Guided Transformer (ASTR) for local feature matching. ASTR models the local consistency and scale variations in a unified coarse-to-fine architecture.
  arXiv  Detail & Related papers  (2023-03-29T12:28:01Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)
• ACFNet: Adaptively-Cooperative Fusion Network for RGB-D Salient Object Detection [0.0]
  We propose an adaptively-cooperative fusion network (ACFNet) with ResinRes structure for salient object detection. For different objects, the features generated by different types of convolution are enhanced or suppressed by the gated mechanism for segmentation optimization. Sufficient experiments conducted on RGB-D SOD datasets illustrate that the proposed network performs favorably against 18 state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-09-10T02:34:27Z)
• Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering [119.88565565454378]
  Unsupervised domain adaptation (UDA) is to learn classification models that make predictions for unlabeled data on a target domain. We propose a hybrid model of Structurally Regularized Deep Clustering, which integrates the regularized discriminative clustering of target data with a generative one. Our proposed H-SRDC outperforms all the existing methods under both the inductive and transductive settings.
  arXiv  Detail & Related papers  (2020-12-08T08:52:00Z)
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels [69.18343801164741]
  partitioned edge learning (PARTEL) implements parameter-server training, a well known distributed learning method, in wireless network. We consider the case of deep neural network (DNN) models which can be trained using PARTEL by introducing some auxiliary variables.
  arXiv  Detail & Related papers  (2020-10-08T15:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.