Resource-Efficient Federated Hyperdimensional Computing
- URL: http://arxiv.org/abs/2306.01339v1
- Date: Fri, 2 Jun 2023 08:07:14 GMT
- Title: Resource-Efficient Federated Hyperdimensional Computing
- Authors: Nikita Zeulin, Olga Galinina, Nageen Himayat, Sergey Andreev
- Abstract summary: In conventional hyperdimensional computing (HDC), training larger models usually results in higher predictive performance but also requires more computational, communication, and energy resources.
A proposed resource-efficient framework alleviates such constraints by training multiple smaller independent HDC sub-models.
Our numerical comparison demonstrates that the proposed framework achieves a comparable or higher predictive performance while consuming less computational and wireless resources.
- Score: 6.778675369739912
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: In conventional federated hyperdimensional computing (HDC), training larger
models usually results in higher predictive performance but also requires more
computational, communication, and energy resources. If the system resources are
limited, one may have to sacrifice the predictive performance by reducing the
size of the HDC model. The proposed resource-efficient federated
hyperdimensional computing (RE-FHDC) framework alleviates such constraints by
training multiple smaller independent HDC sub-models and refining the
concatenated HDC model using the proposed dropout-inspired procedure. Our
numerical comparison demonstrates that the proposed framework achieves a
comparable or higher predictive performance while consuming less computational
and wireless resources than the baseline federated HDC implementation.
Related papers
- DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [70.91804882618243]
This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks.
We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge.
Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
arXiv Detail & Related papers (2025-02-18T02:37:26Z) - HD-CB: The First Exploration of Hyperdimensional Computing for Contextual Bandits Problems [0.6377289349842638]
This work introduces the Hyperdimensional Contextual Bandits (HD-CB)
HD-CB is the first exploration of HDC to model and automate sequential decision-making problems.
It consistently achieves competitive or superior performance compared to traditional linear CB algorithms.
arXiv Detail & Related papers (2025-01-28T11:28:09Z) - Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.
We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
arXiv Detail & Related papers (2024-11-30T10:56:30Z) - Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared
Pre-trained Language Models [109.06052781040916]
We introduce a technique to enhance the inference efficiency of parameter-shared language models.
We also propose a simple pre-training technique that leads to fully or partially shared models.
Results demonstrate the effectiveness of our methods on both autoregressive and autoencoding PLMs.
arXiv Detail & Related papers (2023-10-19T15:13:58Z) - A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical
Computation Offloading [62.34538208323411]
We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs)
MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
arXiv Detail & Related papers (2023-09-02T11:01:16Z) - Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
We take a closer theoretical look at Independent Subnetwork Training (IST)
IST is a recently proposed and highly effective technique for solving the aforementioned problems.
We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
arXiv Detail & Related papers (2023-06-28T18:14:22Z) - CC-FedAvg: Computationally Customized Federated Averaging [11.687451505965655]
Federated learning (FL) is an emerging paradigm to train model with distributed data from numerous Internet of Things (IoT) devices.
We propose a strategy for estimating local models without computationally intensive iterations.
We show that CC-FedAvg has the same convergence rate and comparable performance as FedAvg without resource constraints.
arXiv Detail & Related papers (2022-12-28T03:32:29Z) - Exploiting Temporal Structures of Cyclostationary Signals for
Data-Driven Single-Channel Source Separation [98.95383921866096]
We study the problem of single-channel source separation (SCSS)
We focus on cyclostationary signals, which are particularly suitable in a variety of application domains.
We propose a deep learning approach using a U-Net architecture, which is competitive with the minimum MSE estimator.
arXiv Detail & Related papers (2022-08-22T14:04:56Z) - Quantized Adaptive Subgradient Algorithms and Their Applications [39.103587572626026]
We propose quantized composite mirror descent adaptive subgradient (QCMD adagrad) and quantized regularized dual average adaptive subgradient (QRDA adagrad) for distributed training.
A quantized gradient-based adaptive learning rate matrix is constructed to achieve a balance between communication costs, accuracy, and model sparsity.
arXiv Detail & Related papers (2022-08-11T04:04:03Z) - A Brain-Inspired Low-Dimensional Computing Classifier for Inference on
Tiny Devices [17.976792694929063]
We propose a low-dimensional computing (LDC) alternative to hyperdimensional computing (HDC)
We map our LDC classifier into a neural equivalent network and optimize our model using a principled training approach.
Our LDC classifier offers an overwhelming advantage over the existing brain-inspired HDC models and is particularly suitable for inference on tiny devices.
arXiv Detail & Related papers (2022-03-09T17:20:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.