BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles

• URL: http://arxiv.org/abs/2310.17742v1
• Date: Thu, 26 Oct 2023 19:30:31 GMT
• Title: BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in Time-series Load Profiles
• Authors: Yi Hu, Kai Ye, Hyeonjin Kim and Ning Lu
• Abstract summary: BERT-PIN is a Bidirectional Representations from Transformers powered Profile Inpainting Network. It recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. We develop and evaluate BERT-PIN using real-world dataset for two applications: MDSs recovery and demand response baseline estimation.
• Score: 10.57410710111382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inspired by the success of the Transformer model in natural language processing and computer vision, this paper introduces BERT-PIN, a Bidirectional Encoder Representations from Transformers (BERT) powered Profile Inpainting Network. BERT-PIN recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. To adopt a standard Transformer model structure for profile inpainting, we segment the load and temperature profiles into line segments, treating each segment as a word and the entire profile as a sentence. We incorporate a top candidates selection process in BERT-PIN, enabling it to produce a sequence of probability distributions, based on which users can generate multiple plausible imputed data sets, each reflecting different confidence levels. We develop and evaluate BERT-PIN using real-world dataset for two applications: multiple MDSs recovery and demand response baseline estimation. Simulation results show that BERT-PIN outperforms the existing methods in accuracy while is capable of restoring multiple MDSs within a longer window. BERT-PIN, served as a pre-trained model, can be fine-tuned for conducting many downstream tasks, such as classification and super resolution.

Related papers

• MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer [66.71930982549028]
  Vision-Language Transformers (VLTs) have shown great success recently, but are accompanied by heavy computation costs. We propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs.
  arXiv  Detail & Related papers  (2024-03-05T14:13:50Z)
• CMFDFormer: Transformer-based Copy-Move Forgery Detection with Continual Learning [52.72888626663642]
  Copy-move forgery detection aims at detecting duplicated regions in a suspected forged image. Deep learning based copy-move forgery detection methods are in the ascendant. We propose a Transformer-style copy-move forgery network named as CMFDFormer. We also provide a novel PCSD continual learning framework to help CMFDFormer handle new tasks.
  arXiv  Detail & Related papers  (2023-11-22T09:27:46Z)
• Breaking the Token Barrier: Chunking and Convolution for Efficient Long Text Classification with BERT [0.0]
  Transformer-based models, specifically BERT, have propelled research in various NLP tasks. BERT models are limited to a maximum token limit of 512 tokens. Consequently, this makes it non-trivial to apply it in a practical setting with long input. We propose a relatively simple extension to vanilla BERT architecture called ChunkBERT that allows finetuning of any pretrained models to perform inference on arbitrarily long text.
  arXiv  Detail & Related papers  (2023-10-31T15:41:08Z)
• CLMFormer: Mitigating Data Redundancy to Revitalize Transformer-based Long-Term Time Series Forecasting System [46.39662315849883]
  Long-term time-series forecasting (LTSF) plays a crucial role in various practical applications. Existing Transformer-based models, such as Fedformer and Informer, often achieve their best performances on validation sets after just a few epochs. We propose a novel approach to address this issue by employing curriculum learning and introducing a memory-driven decoder.
  arXiv  Detail & Related papers  (2022-07-16T04:05:15Z)
• Transferring BERT-like Transformers' Knowledge for Authorship Verification [8.443350618722562]
  We study the effectiveness of several BERT-like transformers for the task of authorship verification. We provide new splits for PAN-2020, where training and test data are sampled from disjoint topics or authors. We show that those splits can enhance the models' capability to transfer knowledge over a new, significantly different dataset.
  arXiv  Detail & Related papers  (2021-12-09T18:57:29Z)
• TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference [54.791572981834435]
  Existing pre-trained language models (PLMs) are often computationally expensive in inference. We propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT. TR-BERT formulates the token reduction process as a multi-step token selection problem and automatically learns the selection strategy via reinforcement learning.
  arXiv  Detail & Related papers  (2021-05-25T02:28:51Z)
• VisBERT: Hidden-State Visualizations for Transformers [66.86452388524886]
  We present VisBERT, a tool for visualizing the contextual token representations within BERT for the task of (multi-hop) Question Answering. VisBERT enables users to get insights about the model's internal state and to explore its inference steps or potential shortcomings.
  arXiv  Detail & Related papers  (2020-11-09T15:37:43Z)
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering [68.40719618351429]
  We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings) to solve Visual Question Answering (VQA) Our approach benefits from processing multimodal data adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together.
  arXiv  Detail & Related papers  (2020-10-27T06:34:14Z)
• Incorporating BERT into Parallel Sequence Decoding with Adapters [82.65608966202396]
  We propose to take two different BERT models as the encoder and decoder respectively, and fine-tune them by introducing simple and lightweight adapter modules. We obtain a flexible and efficient model which is able to jointly leverage the information contained in the source-side and target-side BERT models. Our framework is based on a parallel sequence decoding algorithm named Mask-Predict considering the bi-directional and conditional independent nature of BERT.
  arXiv  Detail & Related papers  (2020-10-13T03:25:15Z)
• What BERT Sees: Cross-Modal Transfer for Visual Question Generation [21.640299110619384]
  We study the visual capabilities of BERT out-of-the-box, by avoiding pre-training made on supplementary data. We introduce BERT-gen, a BERT-based architecture for text generation, able to leverage on either mono- or multi- modal representations.
  arXiv  Detail & Related papers  (2020-02-25T12:44:36Z)
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval [11.923682816611716]
  We present TwinBERT model for effective and efficient retrieval. It has twin-structured BERT-like encoders to represent query and document respectively. It allows document embeddings to be pre-computed offline and cached in memory.
  arXiv  Detail & Related papers  (2020-02-14T22:44:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.