InjectTST: A Transformer Method of Injecting Global Information into Independent Channels for Long Time Series Forecasting

• URL: http://arxiv.org/abs/2403.02814v1
• Date: Tue, 5 Mar 2024 09:33:36 GMT
• Title: InjectTST: A Transformer Method of Injecting Global Information into Independent Channels for Long Time Series Forecasting
• Abstract summary: We propose an injection method for global information into channel-independent Transformer, InjectTST. Experiments indicate that InjectTST can achieve stable improvement compared with state-of-the-art models.
• Score: 28.138548907887667
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer has become one of the most popular architectures for multivariate time series (MTS) forecasting. Recent Transformer-based MTS models generally prefer channel-independent structures with the observation that channel independence can alleviate noise and distribution drift issues, leading to more robustness. Nevertheless, it is essential to note that channel dependency remains an inherent characteristic of MTS, carrying valuable information. Designing a model that incorporates merits of both channel-independent and channel-mixing structures is a key to further improvement of MTS forecasting, which poses a challenging conundrum. To address the problem, an injection method for global information into channel-independent Transformer, InjectTST, is proposed in this paper. Instead of designing a channel-mixing model directly, we retain the channel-independent backbone and gradually inject global information into individual channels in a selective way. A channel identifier, a global mixing module and a self-contextual attention module are devised in InjectTST. The channel identifier can help Transformer distinguish channels for better representation. The global mixing module produces cross-channel global information. Through the self-contextual attention module, the independent channels can selectively concentrate on useful global information without robustness degradation, and channel mixing is achieved implicitly. Experiments indicate that InjectTST can achieve stable improvement compared with state-of-the-art models.

Related papers

• Channel-Free Human Activity Recognition via Inductive-Bias-Aware Fusion Design for Heterogeneous IoT Sensor Environments [0.0]
  Human activity recognition (HAR) in Internet of Things (IoT) environments must cope with heterogeneous sensor settings.<n>We investigate strict channel-free HAR, in which a single shared model performs inference without assuming a fixed number, order, or semantic arrangement of input channels.<n>We propose a channel-free HAR framework that combines channel-wise encoding with a shared encoder, metadata-conditioned late fusion, and joint optimization of channel-level and fused predictions.
  arXiv  Detail & Related papers  (2026-04-23T07:32:36Z)
• DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images [9.507520646516719]
  Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations.<n>Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space.<n>We propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA)
  arXiv  Detail & Related papers  (2026-03-15T08:31:34Z)
• Routing Channel-Patch Dependencies in Time Series Forecasting with Graph Spectral Decomposition [44.75749696286735]
  We propose a generic plugin xCPD that can adaptively model the channel-patch dependencies from the perspective of graph spectral decomposition.<n>xCPD can be seamlessly integrated on top of existing CI and CD forecasting models, consistently enhancing both accuracy and generalization across benchmarks.
  arXiv  Detail & Related papers  (2026-03-14T02:18:17Z)
• Context Video Semantic Transmission with Variable Length and Rate Coding over MIMO Channels [49.624608869195065]
  We propose the context video semantic transmission (CVST) framework for wireless video transmission.<n>We learn a context-channel correlation map to explicitly formulate the relationships between feature groups and multiple input multiple output (MIMO) subchannels.<n>We demonstrate substantial performance gains over various standardized separated coding methods and recent wireless video semantic communication approaches.
  arXiv  Detail & Related papers  (2025-12-23T10:48:43Z)
• Flow matching-based generative models for MIMO channel estimation [13.894304144975552]
  Diffusion model (DM)-based channel estimation has shown potential in high-precision channel state information (CSI) acquisition.<n>We propose a novel flow matching (FM)-based generative model for multiple-input multiple-output (MIMO) channel estimation.<n>It achieves superior channel estimation accuracy under different channel conditions.
  arXiv  Detail & Related papers  (2025-11-14T04:05:23Z)
• MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification [9.624687648181231]
  We propose MTM, a multi-scale token mixing transformer for the classification of IMTS.<n>We find that the channel-wise asynchrony can be alleviated by down-sampling the time series to coarser timescales.<n>We also propose a novel channel-wise token mixing mechanism which proactively chooses important tokens from one channel and mixes them with other channels.
  arXiv  Detail & Related papers  (2025-09-22T14:03:02Z)
• Few-Shot Object Detection via Spatial-Channel State Space Model [9.644454618045133]
  Current methods may struggle to accurately extract effective features from each channel.<n>We propose a Spatial-Channel State Space Modeling (SCSM) module for spatial-channel state modeling.<n>SCSM module highlights the effective patterns and rectifies those ineffective ones in feature channels.
  arXiv  Detail & Related papers  (2025-07-21T07:08:19Z)
• RelUNet: Relative Channel Fusion U-Net for Multichannel Speech Enhancement [25.878204820665516]
  Multi-channel speech enhancement models, in particular those based on the U-Net architecture, demonstrate promising performance and generalization potential. We propose a novel modification of these models by incorporating relative information from the outset, where each channel is processed in conjunction with a reference channel through stacking. This input strategy exploits comparative differences to adaptively fuse information between channels, thereby capturing crucial spatial information and enhancing the overall performance.
  arXiv  Detail & Related papers  (2024-10-07T13:19:10Z)
• SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion [59.96233305733875]
  Time series forecasting plays a crucial role in various fields such as finance, traffic management, energy, and healthcare. Several methods utilize mechanisms like attention or mixer to address this by capturing channel correlations. This paper presents an efficient-based model, the Series-cOre Fused Time Series forecaster (SOFTS)
  arXiv  Detail & Related papers  (2024-04-22T14:06:35Z)
• From Similarity to Superiority: Channel Clustering for Time Series Forecasting [61.96777031937871]
  We develop a novel and adaptable Channel Clustering Module ( CCM) CCM dynamically groups channels characterized by intrinsic similarities and leverages cluster information instead of individual channel identities. CCM can boost the performance of CI and CD models by an average margin of 2.4% and 7.2% on long-term and short-term forecasting, respectively.
  arXiv  Detail & Related papers  (2024-03-31T02:46:27Z)
• MCformer: Multivariate Time Series Forecasting with Mixed-Channels Transformer [8.329947472853029]
  Channel Independence (CI) strategy treats all channels as a single channel, expanding the dataset. Mixed Channels strategy combines the data expansion advantages of the CI strategy with the ability to counteract inter-channel correlation forgetting. Model blends a specific number of channels, leveraging an attention mechanism to effectively capture inter-channel correlation information.
  arXiv  Detail & Related papers  (2024-03-14T09:43:07Z)
• CSformer: Combining Channel Independence and Mixing for Robust Multivariate Time Series Forecasting [3.6814181034608664]
  We propose a strategy of channel independence followed by mixing in time series analysis. We introduce CSformer, a novel framework featuring a two-stage multiheaded self-attention mechanism. Our framework effectively incorporates sequence and channel adapters, significantly improving the model's ability to identify important information.
  arXiv  Detail & Related papers  (2023-12-11T09:10:38Z)
• CB-HVTNet: A channel-boosted hybrid vision transformer network for lymphocyte assessment in histopathological images [0.40777876591043144]
  We propose a Channel Boosted Hybrid Vision Transformer (CB HVT) that uses transfer learning to generate boosted channels and employs both transformers and CNNs to analyse lymphocytes in medical images. CB HVT comprises five modules, including a channel generation module, channel exploitation module, channel merging module, region-aware module, and a detection and segmentation head. The results show that CB HVT outperformed other state of the art detection models, and has good generalization ability, demonstrating its value as a tool for pathologists.
  arXiv  Detail & Related papers  (2023-05-16T06:40:04Z)
• Model-Driven Deep Learning Based Channel Estimation and Feedback for Millimeter-Wave Massive Hybrid MIMO Systems [61.78590389147475]
  This paper proposes a model-driven deep learning (MDDL)-based channel estimation and feedback scheme for millimeter-wave (mmWave) systems. To reduce the uplink pilot overhead for estimating the high-dimensional channels from a limited number of radio frequency (RF) chains, we propose to jointly train the phase shift network and the channel estimator as an auto-encoder. Numerical results show that the proposed MDDL-based channel estimation and feedback scheme outperforms the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-04-22T13:34:53Z)
• Speaker Representation Learning using Global Context Guided Channel and Time-Frequency Transformations [67.18006078950337]
  We use the global context information to enhance important channels and recalibrate salient time-frequency locations. The proposed modules, together with a popular ResNet based model, are evaluated on the VoxCeleb1 dataset.
  arXiv  Detail & Related papers  (2020-09-02T01:07:29Z)
• Single Image Super-Resolution via a Holistic Attention Network [87.42409213909269]
  We propose a new holistic attention network (HAN) to model the holistic interdependencies among layers, channels, and positions. The proposed HAN adaptively emphasizes hierarchical features by considering correlations among layers. Experiments demonstrate that the proposed HAN performs favorably against the state-of-the-art single image super-resolution approaches.
  arXiv  Detail & Related papers  (2020-08-20T04:13:15Z)
• Data-Driven Symbol Detection via Model-Based Machine Learning [117.58188185409904]
  We review a data-driven framework to symbol detection design which combines machine learning (ML) and model-based algorithms. In this hybrid approach, well-known channel-model-based algorithms are augmented with ML-based algorithms to remove their channel-model-dependence. Our results demonstrate that these techniques can yield near-optimal performance of model-based algorithms without knowing the exact channel input-output statistical relationship.
  arXiv  Detail & Related papers  (2020-02-14T06:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.