Soft Sensing Transformer: Hundreds of Sensors are Worth a Single Word
- URL: http://arxiv.org/abs/2111.05973v1
- Date: Wed, 10 Nov 2021 22:31:32 GMT
- Title: Soft Sensing Transformer: Hundreds of Sensors are Worth a Single Word
- Authors: Chao Zhang, Jaswanth Yella, Yu Huang, Xiaoye Qian, Sergei Petrov,
Andrey Rzhetsky, Sthitie Bom
- Abstract summary: We demonstrate the challenges and effectiveness of modeling industrial big data by a Soft Sensing Transformer model.
We observe the similarity of a sentence structure to the sensor readings and process the multi-dimensional sensor readings in a time series in a similar manner of sentences in natural language.
The results show that transformer model outperforms the benchmark models in soft sensing field based on auto-encoder and long short-term memory (LSTM) models.
- Score: 4.829772176792801
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With the rapid development of AI technology in recent years, there have been
many studies with deep learning models in soft sensing area. However, the
models have become more complex, yet, the data sets remain limited: researchers
are fitting million-parameter models with hundreds of data samples, which is
insufficient to exercise the effectiveness of their models and thus often fail
to perform when implemented in industrial applications. To solve this
long-lasting problem, we are providing large scale, high dimensional time
series manufacturing sensor data from Seagate Technology to the public. We
demonstrate the challenges and effectiveness of modeling industrial big data by
a Soft Sensing Transformer model on these data sets. Transformer is used
because, it has outperformed state-of-the-art techniques in Natural Language
Processing, and since then has also performed well in the direct application to
computer vision without introduction of image-specific inductive biases. We
observe the similarity of a sentence structure to the sensor readings and
process the multi-variable sensor readings in a time series in a similar manner
of sentences in natural language. The high-dimensional time-series data is
formatted into the same shape of embedded sentences and fed into the
transformer model. The results show that transformer model outperforms the
benchmark models in soft sensing field based on auto-encoder and long
short-term memory (LSTM) models. To the best of our knowledge, we are the first
team in academia or industry to benchmark the performance of original
transformer model with large-scale numerical soft sensing data.
Related papers
- A deep latent variable model for semi-supervised multi-unit soft sensing in industrial processes [0.0]
We introduce a deep latent variable model for semi-supervised multi-unit soft sensing.
This hierarchical, generative model is able to jointly model different units, as well as learning from both labeled and unlabeled data.
We show that by combining semi-supervised and multi-task learning, the proposed model achieves superior results.
arXiv Detail & Related papers (2024-07-18T09:13:22Z) - A Predictive Model Based on Transformer with Statistical Feature Embedding in Manufacturing Sensor Dataset [2.07180164747172]
This study proposes a novel predictive model based on the Transformer, utilizing statistical feature embedding and window positional encoding.
The model's performance is evaluated in two problems: fault detection and virtual metrology, showing superior results compared to baseline models.
The results support the model's applicability across various manufacturing industries, demonstrating its potential for enhancing process management and yield.
arXiv Detail & Related papers (2024-07-09T08:59:27Z) - A Transformer-based Framework For Multi-variate Time Series: A Remaining
Useful Life Prediction Use Case [4.0466311968093365]
This work proposed an encoder-transformer architecture-based framework for time series prediction.
We validated the effectiveness of the proposed framework on all four sets of the C-MAPPS benchmark dataset.
To enable the model awareness of the initial stages of the machine life and its degradation path, a novel expanding window method was proposed.
arXiv Detail & Related papers (2023-08-19T02:30:35Z) - Emergent Agentic Transformer from Chain of Hindsight Experience [96.56164427726203]
We show that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches.
This is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches.
arXiv Detail & Related papers (2023-05-26T00:43:02Z) - Robust representations of oil wells' intervals via sparse attention
mechanism [2.604557228169423]
We introduce the class of efficient Transformers named Regularized Transformers (Reguformers)
The focus in our experiments is on oil&gas data, namely, well logs.
To evaluate our models for such problems, we work with an industry-scale open dataset consisting of well logs of more than 20 wells.
arXiv Detail & Related papers (2022-12-29T09:56:33Z) - Advancing Plain Vision Transformer Towards Remote Sensing Foundation
Model [97.9548609175831]
We resort to plain vision transformers with about 100 million parameters and make the first attempt to propose large vision models customized for remote sensing tasks.
Specifically, to handle the large image size and objects of various orientations in RS images, we propose a new rotated varied-size window attention.
Experiments on detection tasks demonstrate the superiority of our model over all state-of-the-art models, achieving 81.16% mAP on the DOTA-V1.0 dataset.
arXiv Detail & Related papers (2022-08-08T09:08:40Z) - Auto-encoder based Model for High-dimensional Imbalanced Industrial Data [6.339700878842761]
We introduce a variance weighted multi-headed auto-encoder classification model that fits well into the high-dimensional and highly imbalanced data.
The model also simultaneously predicts multiple outputs by exploiting output-supervised representation learning and multi-task weighting.
arXiv Detail & Related papers (2021-08-04T14:34:59Z) - STAR: Sparse Transformer-based Action Recognition [61.490243467748314]
This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data.
Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference.
arXiv Detail & Related papers (2021-07-15T02:53:11Z) - Visformer: The Vision-friendly Transformer [105.52122194322592]
We propose a new architecture named Visformer, which is abbreviated from the Vision-friendly Transformer'
With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy.
arXiv Detail & Related papers (2021-04-26T13:13:03Z) - Spatiotemporal Transformer for Video-based Person Re-identification [102.58619642363958]
We show that, despite the strong learning ability, the vanilla Transformer suffers from an increased risk of over-fitting.
We propose a novel pipeline where the model is pre-trained on a set of synthesized video data and then transferred to the downstream domains.
The derived algorithm achieves significant accuracy gain on three popular video-based person re-identification benchmarks.
arXiv Detail & Related papers (2021-03-30T16:19:27Z) - Long Range Arena: A Benchmark for Efficient Transformers [115.1654897514089]
Long-rangearena benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens.
We systematically evaluate ten well-established long-range Transformer models on our newly proposed benchmark suite.
arXiv Detail & Related papers (2020-11-08T15:53:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.