Modeling Accurate Human Activity Recognition for Embedded Devices Using
Multi-level Distillation
- URL: http://arxiv.org/abs/2107.07331v1
- Date: Tue, 6 Jul 2021 09:01:41 GMT
- Title: Modeling Accurate Human Activity Recognition for Embedded Devices Using
Multi-level Distillation
- Authors: Runze Chen and Haiyong Luo and Fang Zhao and Xuechun Meng and Zhiqing
Xie and Yida Zhu
- Abstract summary: Human activity recognition (HAR) based on IMU sensors is an essential domain in ubiquitous computing.
We propose a plug-and-play HAR modeling pipeline with multi-level distillation to build deep convolutional HAR models with native support of embedded devices.
We compare the performance of accuracy, F1 macro score, and energy cost on the embedded platform of various state-of-the-art HAR frameworks with a MobileNet V3 model built by SMLDist.
- Score: 5.746224188845082
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Human activity recognition (HAR) based on IMU sensors is an essential domain
in ubiquitous computing. Because of the improving trend to deploy artificial
intelligence into IoT devices or smartphones, more researchers design the HAR
models for embedded devices. We propose a plug-and-play HAR modeling pipeline
with multi-level distillation to build deep convolutional HAR models with
native support of embedded devices. SMLDist consists of stage distillation,
memory distillation, and logits distillation, which covers all the information
flow of the deep models. Stage distillation constrains the learning direction
of the intermediate features. Memory distillation teaches the student models
how to explain and store the inner relationship between high-dimensional
features based on Hopfield networks. Logits distillation constructs distilled
logits by a smoothed conditional rule to keep the probable distribution and
improve the correctness of the soft target. We compare the performance of
accuracy, F1 macro score, and energy cost on the embedded platform of various
state-of-the-art HAR frameworks with a MobileNet V3 model built by SMLDist. The
produced model has well balance with robustness, efficiency, and accuracy.
SMLDist can also compress the models with minor performance loss in an equal
compression rate than other state-of-the-art knowledge distillation methods on
seven public datasets.
Related papers
- FSD-BEV: Foreground Self-Distillation for Multi-view 3D Object Detection [33.225938984092274]
We propose a Foreground Self-Distillation (FSD) scheme that effectively avoids the issue of distribution discrepancies.
We also design two Point Cloud Intensification ( PCI) strategies to compensate for the sparsity of point clouds.
We develop a Multi-Scale Foreground Enhancement (MSFE) module to extract and fuse multi-scale foreground features.
arXiv Detail & Related papers (2024-07-14T09:39:44Z) - Promoting CNNs with Cross-Architecture Knowledge Distillation for Efficient Monocular Depth Estimation [4.242540533823568]
Transformer models are usually computationally-expensive, and their effectiveness in light-weight models are limited compared to convolutions.
We propose a cross-architecture knowledge distillation method for MDE, dubbed DisDepth, to enhance efficient CNN models with the supervision of state-of-the-art transformer models.
Our method achieves significant improvements on various efficient backbones, showcasing its potential for efficient monocular depth estimation.
arXiv Detail & Related papers (2024-04-25T07:55:47Z) - Progressive Volume Distillation with Active Learning for Efficient NeRF Architecture Conversion [27.389511043400635]
Neural Fields (NeRF) have been widely adopted as practical and versatile representations for 3D scenes.
We propose Progressive Volume Distillation with Active Learning (PVD-AL), a systematic distillation method.
PVD-AL decomposes each structure into two parts and progressively performs distillation from shallower to deeper volume representation.
arXiv Detail & Related papers (2023-04-08T13:59:18Z) - SimDistill: Simulated Multi-modal Distillation for BEV 3D Object
Detection [56.24700754048067]
Multi-view camera-based 3D object detection has become popular due to its low cost, but accurately inferring 3D geometry solely from camera data remains challenging.
We propose a Simulated multi-modal Distillation (SimDistill) method by carefully crafting the model architecture and distillation strategy.
Our SimDistill can learn better feature representations for 3D object detection while maintaining a cost-effective camera-only deployment.
arXiv Detail & Related papers (2023-03-29T16:08:59Z) - Generic-to-Specific Distillation of Masked Autoencoders [119.21281960831651]
We propose generic-to-specific distillation (G2SD) to tap the potential of small ViT models under the supervision of large models pre-trained by masked autoencoders.
With G2SD, the vanilla ViT-Small model achieves 98.7%, 98.1% and 99.3% the performance of its teacher for image classification, object detection, and semantic segmentation.
arXiv Detail & Related papers (2023-02-28T17:13:14Z) - Directed Acyclic Graph Factorization Machines for CTR Prediction via
Knowledge Distillation [65.62538699160085]
We propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation.
KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments.
arXiv Detail & Related papers (2022-11-21T03:09:42Z) - Multimodal Transformer Distillation for Audio-Visual Synchronization [53.237653873618754]
This paper proposed an MTDVocaLiST model, which is trained by our proposed multimodal Transformer distillation (MTD) loss.
MTDVocaLiST reduces the model size of VocaLiST by 83.52%, yet still maintaining similar performance.
arXiv Detail & Related papers (2022-10-27T15:53:38Z) - Efficient Federated Learning for AIoT Applications Using Knowledge
Distillation [2.5892786553124085]
Federated Learning (FL) trains a central model with decentralized data without compromising user privacy.
Traditional FL suffers from model inaccuracy since it trains local models using hard labels of data.
This paper presents a novel Distillation-based Federated Learning architecture that enables efficient and accurate FL for AIoT applications.
arXiv Detail & Related papers (2021-11-29T06:40:42Z) - Self-Feature Regularization: Self-Feature Distillation Without Teacher
Models [0.0]
Self-Feature Regularization(SFR) is proposed, which uses features in the deep layers to supervise feature learning in the shallow layers.
We firstly use generalization-l2 loss to match local features and a many-to-one approach to distill more intensively in the channel dimension.
arXiv Detail & Related papers (2021-03-12T15:29:00Z) - Physics-Integrated Variational Autoencoders for Robust and Interpretable
Generative Modeling [86.9726984929758]
We focus on the integration of incomplete physics models into deep generative models.
We propose a VAE architecture in which a part of the latent space is grounded by physics.
We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
arXiv Detail & Related papers (2021-02-25T20:28:52Z) - Ensemble Distillation for Robust Model Fusion in Federated Learning [72.61259487233214]
Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model.
In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side.
We propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients.
arXiv Detail & Related papers (2020-06-12T14:49:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.