Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging

• URL: http://arxiv.org/abs/2306.01176v2
• Date: Thu, 20 Mar 2025 00:27:01 GMT
• Title: Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging
• Authors: Jiamian Wang, Zongliang Wu, Yulun Zhang, Xin Yuan, Tao Lin, Zhiqiang Tao,
• Abstract summary: We propose a Federated Hardware-Prompt learning (FedHP) framework to cooperatively optimize snapshot compressive imaging systems.<n>FedHP learns a hardware-conditioned prompter to align inconsistent data distribution across clients, serving as an indicator of the data inconsistency among different hardware.<n>Experiments demonstrate that the proposed FedHP coordinates the pre-trained model to multiple hardware configurations, outperforming prevalent FL frameworks for 0.35dB.
• Score: 51.65127848056702
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing reconstruction models in snapshot compressive imaging systems (SCI) are trained with a single well-calibrated hardware instance, making their performance vulnerable to hardware shifts and limited in adapting to multiple hardware configurations. To facilitate cross-hardware learning, previous efforts attempt to directly collect multi-hardware data and perform centralized training, which is impractical due to severe user data privacy concerns and hardware heterogeneity across different platforms/institutions. In this study, we explicitly consider data privacy and heterogeneity in cooperatively optimizing SCI systems by proposing a Federated Hardware-Prompt learning (FedHP) framework. Rather than mitigating the client drift by rectifying the gradients, which only takes effect on the learning manifold but fails to solve the heterogeneity rooted in the input data space, FedHP learns a hardware-conditioned prompter to align inconsistent data distribution across clients, serving as an indicator of the data inconsistency among different hardware (e.g., coded apertures). Extensive experimental results demonstrate that the proposed FedHP coordinates the pre-trained model to multiple hardware configurations, outperforming prevalent FL frameworks for 0.35dB under challenging heterogeneous settings. Moreover, a Snapshot Spectral Heterogeneous Dataset has been built upon multiple practical SCI systems. Data and code are aveilable at https://github.com/Jiamian-Wang/FedHP-Snapshot-Compressive-Imaging

Related papers

• Federated Learning of Low-Rank One-Shot Image Detection Models in Edge Devices with Scalable Accuracy and Compute Complexity [5.820612543019548]
  LoRa-FL is designed for training low-rank one-shot image detection models deployed on edge devices. By incorporating low-rank adaptation techniques into one-shot detection architectures, our method significantly reduces both computational and communication overhead.
  arXiv  Detail & Related papers  (2025-04-23T08:40:44Z)
• FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors [50.131271229165165]
  Federated Learning (FL) has emerged as a promising framework for distributed machine learning. Data heterogeneity resulting from differences across user behaviors, preferences, and device characteristics poses a significant challenge for federated learning. We propose Adaptive Weight Aggregation (FedAWA), a novel method that adaptively adjusts aggregation weights based on client vectors during the learning process.
  arXiv  Detail & Related papers  (2025-03-20T04:49:40Z)
• Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers [3.2492319522383717]
  Contrastive Language-Image Pre-training (CLIP) has attracted a surge of attention for its superior zero-shot performance and excellent transferability to downstream tasks. However, training such large-scale models usually requires substantial computation and storage, which poses barriers for general users with consumer-level computers.
  arXiv  Detail & Related papers  (2024-11-22T08:17:46Z)
• Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis [50.18156030818883]
  Anomaly and missing data constitute a thorny problem in industrial applications. Deep learning enabled anomaly detection has emerged as a critical direction. The data collected in edge devices contain user privacy.
  arXiv  Detail & Related papers  (2024-11-06T15:38:31Z)
• InfRS: Incremental Few-Shot Object Detection in Remote Sensing Images [11.916941756499435]
  In this paper, we explore the intricate task of incremental few-shot object detection in remote sensing images. We introduce a pioneering fine-tuning-based technique, termed InfRS, designed to facilitate the incremental learning of novel classes. We develop a prototypical calibration strategy based on the Wasserstein distance to mitigate the catastrophic forgetting problem.
  arXiv  Detail & Related papers  (2024-05-18T13:39:50Z)
• Deepfake Sentry: Harnessing Ensemble Intelligence for Resilient Detection and Generalisation [0.8796261172196743]
  We propose a proactive and sustainable deepfake training augmentation solution. We employ a pool of autoencoders that mimic the effect of the artefacts introduced by the deepfake generator models. Experiments reveal that our proposed ensemble autoencoder-based data augmentation learning approach offers improvements in terms of generalisation.
  arXiv  Detail & Related papers  (2024-03-29T19:09:08Z)
• Depth-agnostic Single Image Dehazing [12.51359372069387]
  We propose a simple yet novel synthetic method to decouple the relationship between haze density and scene depth, by which a depth-agnostic dataset (DA-HAZE) is generated. Experiments indicate that models trained on DA-HAZE achieve significant improvements on real-world benchmarks, with less discrepancy between SOTS and DA-SOTS. We revisit the U-Net-based architectures for dehazing, in which dedicatedly designed blocks are incorporated.
  arXiv  Detail & Related papers  (2024-01-14T06:33:11Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneity [56.82825745165945]
  Federated Learning (FL) enables training a global model without sharing the decentralized raw data stored on multiple devices to protect data privacy. We propose a hierarchical synchronous FL framework, i.e., FedHiSyn, to tackle the problems of straggler effects and outdated models. We evaluate the proposed framework based on MNIST, EMNIST, CIFAR10 and CIFAR100 datasets and diverse heterogeneous settings of devices.
  arXiv  Detail & Related papers  (2022-06-21T17:23:06Z)
• Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging [142.11622043078867]
  We propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST) for HSI reconstruction.
  arXiv  Detail & Related papers  (2022-05-20T11:37:44Z)
• Towards Tailored Models on Private AIoT Devices: Federated Direct Neural Architecture Search [22.69123714900226]
  We propose a Federated Direct Neural Architecture Search (FDNAS) framework that allows for hardware-friendly NAS from non- IID data across devices. Experiments on non-IID datasets have shown the state-of-the-art accuracy-efficiency trade-offs achieved by the proposed solution.
  arXiv  Detail & Related papers  (2022-02-23T13:10:01Z)
• Stochastic Coded Federated Learning with Convergence and Privacy Guarantees [8.2189389638822]
  Federated learning (FL) has attracted much attention as a privacy-preserving distributed machine learning framework. This paper proposes a coded federated learning framework, namely coded federated learning (SCFL) to mitigate the straggler issue. We characterize the privacy guarantee by the mutual information differential privacy (MI-DP) and analyze the convergence performance in federated learning.
  arXiv  Detail & Related papers  (2022-01-25T04:43:29Z)
• Robust Semi-supervised Federated Learning for Images Automatic Recognition in Internet of Drones [57.468730437381076]
  We present a Semi-supervised Federated Learning (SSFL) framework for privacy-preserving UAV image recognition. There are significant differences in the number, features, and distribution of local data collected by UAVs using different camera modules. We propose an aggregation rule based on the frequency of the client's participation in training, namely the FedFreq aggregation rule.
  arXiv  Detail & Related papers  (2022-01-03T16:49:33Z)
• Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition [124.80263629921498]
  We propose Pixel Distillation that extends knowledge distillation into the input level while simultaneously breaking architecture constraints. Such a scheme can achieve flexible cost control for deployment, as it allows the system to adjust both network architecture and image quality according to the overall requirement of resources.
  arXiv  Detail & Related papers  (2021-12-17T14:31:40Z)
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning [53.73083199055093]
  We show that attention-based architectures (e.g., Transformers) are fairly robust to distribution shifts. Our experiments show that replacing convolutional networks with Transformers can greatly reduce catastrophic forgetting of previous devices.
  arXiv  Detail & Related papers  (2021-06-10T21:04:18Z)
• Fast Federated Learning in the Presence of Arbitrary Device Unavailability [26.368873771739715]
  Federated Learning (FL) coordinates heterogeneous devices to collaboratively train a shared model while preserving user privacy. One challenge arises when devices drop out of the training process beyond the central server. We propose Im Federated Apatientaging (MIFA) to solve this problem.
  arXiv  Detail & Related papers  (2021-06-08T07:46:31Z)
• Federated Doubly Stochastic Kernel Learning for Vertically Partitioned Data [93.76907759950608]
  We propose a doubly kernel learning algorithm for vertically partitioned data. We show that FDSKL is significantly faster than state-of-the-art federated learning methods when dealing with kernels.
  arXiv  Detail & Related papers  (2020-08-14T05:46:56Z)
• FPGA-Based Hardware Accelerator of Homomorphic Encryption for Efficient Federated Learning [9.733675923979108]
  Federated learning tends to utilize various privacy preserving mechanisms to protect the transferred intermediate data. Maintaining accuracy and security more efficiently has been a key problem of federated learning. Our framework implements the representative Paillier homomorphic cryptosystem with high level synthesis for flexibility and portability.
  arXiv  Detail & Related papers  (2020-07-21T01:59:58Z)
• FedCD: Improving Performance in non-IID Federated Learning [0.0]
  Federated learning has been widely applied to enable decentralized devices, which each have their own local data, to learn a shared model. We present a novel approach, FedCD, which clones and deletes models to dynamically group devices with similar data.
  arXiv  Detail & Related papers  (2020-06-17T03:45:25Z)
• Learning End-to-End Lossy Image Compression: A Benchmark [90.35363142246806]
  We first conduct a comprehensive literature survey of learned image compression methods. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance.
  arXiv  Detail & Related papers  (2020-02-10T13:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.