Efficient Localness Transformer for Smart Sensor-Based Energy
Disaggregation
- URL: http://arxiv.org/abs/2203.16537v1
- Date: Tue, 29 Mar 2022 22:58:39 GMT
- Title: Efficient Localness Transformer for Smart Sensor-Based Energy
Disaggregation
- Authors: Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, Dong Wang
- Abstract summary: We propose an efficient localness transformer for non-intrusive load monitoring (NILM)
Specifically, we leverage normalization functions and switch the order of matrix multiplication to approximate self-attention.
We demonstrate the efficiency and effectiveness of the the proposed ELTransformer with considerable improvements compared to state-of-the-art baselines.
- Score: 8.828396559882954
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Modern smart sensor-based energy management systems leverage non-intrusive
load monitoring (NILM) to predict and optimize appliance load distribution in
real-time. NILM, or energy disaggregation, refers to the decomposition of
electricity usage conditioned on the aggregated power signals (i.e., smart
sensor on the main channel). Based on real-time appliance power prediction
using sensory technology, energy disaggregation has great potential to increase
electricity efficiency and reduce energy expenditure. With the introduction of
transformer models, NILM has achieved significant improvements in predicting
device power readings. Nevertheless, transformers are less efficient due to
O(l^2) complexity w.r.t. sequence length l. Moreover, transformers can fail to
capture local signal patterns in sequence-to-point settings due to the lack of
inductive bias in local context. In this work, we propose an efficient
localness transformer for non-intrusive load monitoring (ELTransformer).
Specifically, we leverage normalization functions and switch the order of
matrix multiplication to approximate self-attention and reduce computational
complexity. Additionally, we introduce localness modeling with sparse local
attention heads and relative position encodings to enhance the model capacity
in extracting short-term local patterns. To the best of our knowledge,
ELTransformer is the first NILM model that addresses computational complexity
and localness modeling in NILM. With extensive experiments and quantitative
analyses, we demonstrate the efficiency and effectiveness of the the proposed
ELTransformer with considerable improvements compared to state-of-the-art
baselines.
Related papers
- ARTEMIS: A Mixed Analog-Stochastic In-DRAM Accelerator for Transformer Neural Networks [2.9699290794642366]
ARTEMIS is a mixed analog-stochastic in-DRAM accelerator for transformer models.
Our analysis indicates that ARTEMIS exhibits at least 3.0x speedup, 1.8x lower energy, and 1.9x better energy efficiency compared to GPU, TPU, CPU, and state-of-the-art PIM transformer hardware accelerators.
arXiv Detail & Related papers (2024-07-17T15:08:14Z) - Residual-based Attention Physics-informed Neural Networks for Efficient Spatio-Temporal Lifetime Assessment of Transformers Operated in Renewable Power Plants [0.6223528900192875]
This article introduces an efficienttemporal model for transformer winding temperature and ageing estimation.
It uses physics-based partial differential equations with data-driven Neural Networks.
It is validated with a distribution transformer operated on a floating photovoltaic power plant.
arXiv Detail & Related papers (2024-05-10T12:48:57Z) - Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics [11.182510067821745]
This study introduces a novel transformer model optimized for large-scale point cloud processing.
Our model integrates local inductive bias and achieves near-linear complexity with hardware-friendly regular operations.
Our findings highlight the superiority of using locality-sensitive hashing (LSH), especially OR & AND-construction LSH, in kernel approximation for large-scale point cloud data.
arXiv Detail & Related papers (2024-02-19T20:48:09Z) - 2-D SSM: A General Spatial Layer for Visual Transformers [79.4957965474334]
A central objective in computer vision is to design models with appropriate 2-D inductive bias.
We leverage an expressive variation of the multidimensional State Space Model.
Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme.
arXiv Detail & Related papers (2023-06-11T09:41:37Z) - Effective Pre-Training Objectives for Transformer-based Autoencoders [97.99741848756302]
We study trade-offs between efficiency, cost and accuracy of Transformer encoders.
We combine features of common objectives and create new effective pre-training approaches.
arXiv Detail & Related papers (2022-10-24T18:39:44Z) - Stabilizing Voltage in Power Distribution Networks via Multi-Agent
Reinforcement Learning with Transformer [128.19212716007794]
We propose a Transformer-based Multi-Agent Actor-Critic framework (T-MAAC) to stabilize voltage in power distribution networks.
In addition, we adopt a novel auxiliary-task training process tailored to the voltage control task, which improves the sample efficiency.
arXiv Detail & Related papers (2022-06-08T07:48:42Z) - Your Transformer May Not be as Powerful as You Expect [88.11364619182773]
We mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions.
We present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is.
We develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions.
arXiv Detail & Related papers (2022-05-26T14:51:30Z) - An Artificial Neural Network-Based Model Predictive Control for
Three-phase Flying Capacitor Multi-Level Inverter [2.3513645401551333]
Model predictive control (MPC) has been used widely in power electronics due to its simple concept, fast dynamic response, and good reference tracking.
It suffers from parametric uncertainties, since it relies on the mathematical model of the system to predict the optimal switching states.
This paper offers a model-free control strategy on the basis of artificial neural networks (ANNs)
arXiv Detail & Related papers (2021-10-15T13:54:08Z) - Efficient pre-training objectives for Transformers [84.64393460397471]
We study several efficient pre-training objectives for Transformers-based models.
We prove that eliminating the MASK token and considering the whole output during the loss are essential choices to improve performance.
arXiv Detail & Related papers (2021-04-20T00:09:37Z) - Massive MIMO As an Extreme Learning Machine [83.12538841141892]
A massive multiple-input multiple-output (MIMO) system with low-resolution analog-to-digital converters (ADCs) forms a natural extreme learning machine (ELM)
By adding random biases to the received signals and optimizing the ELM output weights, the system can effectively tackle hardware impairments.
arXiv Detail & Related papers (2020-07-01T04:15:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.