Reverse Ordering Techniques for Attention-Based Channel Prediction

• URL: http://arxiv.org/abs/2302.00341v2
• Date: Thu, 11 May 2023 10:50:56 GMT
• Title: Reverse Ordering Techniques for Attention-Based Channel Prediction
• Authors: Valentina Rizzello, Benedikt B\"ock, Michael Joham, Wolfgang Utschick
• Abstract summary: This work aims to predict channels in wireless communication systems based on noisy observations. Models are adapted from natural language processing to tackle the complex challenge of channel prediction.
• Score: 11.630651920572221
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work aims to predict channels in wireless communication systems based on noisy observations, utilizing sequence-to-sequence models with attention (Seq2Seq-attn) and transformer models. Both models are adapted from natural language processing to tackle the complex challenge of channel prediction. Additionally, a new technique called reverse positional encoding is introduced in the transformer model to improve the robustness of the model against varying sequence lengths. Similarly, the encoder outputs of the Seq2Seq-attn model are reversed before applying attention. Simulation results demonstrate that the proposed ordering techniques allow the models to better capture the relationships between the channel snapshots within the sequence, irrespective of the sequence length, as opposed to existing methods.

Related papers

• Sequential Order-Robust Mamba for Time Series Forecasting [5.265578815577529]
  Mamba has emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data. We propose SOR-Mamba, a TS forecasting method that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders. We also introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.
  arXiv  Detail & Related papers  (2024-10-30T18:05:22Z)
• Texture, Shape, Order, and Relation Matter: A New Transformer Design for Sequential DeepFake Detection [81.04969838138634]
  Sequential DeepFake detection is an emerging task that predicts the manipulation sequence in order.<n>This paper describes a new Transformer design, called TSOM, by exploring three perspectives: Texture, Shape, and Order of Manipulations.<n>We conduct extensive experiments in comparison with several state-of-the-art methods, demonstrating the superiority of our method.
  arXiv  Detail & Related papers  (2024-04-22T04:47:52Z)
• Non-autoregressive Sequence-to-Sequence Vision-Language Models [63.77614880533488]
  We propose a parallel decoding sequence-to-sequence vision-language model that marginalizes over multiple inference paths in the decoder. The model achieves performance on-par with its state-of-the-art autoregressive counterpart, but is faster at inference time.
  arXiv  Detail & Related papers  (2024-03-04T17:34:59Z)
• Toeplitz Neural Network for Sequence Modeling [46.04964190407727]
  We show that a Toeplitz matrix-vector production trick can reduce the space-time complexity of the sequence modeling to log linear. A lightweight sub-network called relative position encoder is proposed to generate relative position coefficients with a fixed budget of parameters. Despite being trained on 512-token sequences, our model can extrapolate input sequence length up to 14K tokens in inference with consistent performance.
  arXiv  Detail & Related papers  (2023-05-08T14:49:01Z)
• STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition [50.064502884594376]
  We study the problem of human action recognition using motion capture (MoCap) sequences. We propose a novel Spatial-Temporal Mesh Transformer (STMT) to directly model the mesh sequences. The proposed method achieves state-of-the-art performance compared to skeleton-based and point-cloud-based models.
  arXiv  Detail & Related papers  (2023-03-31T16:19:27Z)
• SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers [50.90457644954857]
  In this work, we apply diffusion models to approach sequence-to-sequence text generation. We propose SeqDiffuSeq, a text diffusion model for sequence-to-sequence generation. Experiment results illustrate the good performance on sequence-to-sequence generation in terms of text quality and inference time.
  arXiv  Detail & Related papers  (2022-12-20T15:16:24Z)
• Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration [85.3323211054274]
  How to properly model the inter-frame relation within the video sequence is an important but unsolved challenge for video restoration (VR) In this work, we propose an unsupervised flow-aligned sequence-to-sequence model (S2SVR) to address this problem. S2SVR shows superior performance in multiple VR tasks, including video deblurring, video super-resolution, and compressed video quality enhancement.
  arXiv  Detail & Related papers  (2022-05-20T14:14:48Z)
• Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding [21.978994865937786]
  The method performs a few refinement steps, where each step shares a transformer decoder that attends to both text features and audio features. We show that, conditioned on hypothesis alignments of a streaming RNN-T model, our method obtains significantly more accurate recognition results than the first-pass RNN-T.
  arXiv  Detail & Related papers  (2021-12-01T01:34:28Z)
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction [86.94309120789396]
  We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks.
  arXiv  Detail & Related papers  (2021-06-06T21:53:54Z)
• Translate Reverberated Speech to Anechoic Ones: Speech Dereverberation with BERT [6.876734825043823]
  Single channel speech dereverberation is considered in this work. Inspired by the recent success of Bidirectional Representations from Transformers (BERT) model in the domain of Natural Language Processing (NLP), we investigate its applicability as backbone sequence model to enhance reverberated speech signal.
  arXiv  Detail & Related papers  (2020-07-16T00:45:27Z)
• DiscreTalk: Text-to-Speech as a Machine Translation Problem [52.33785857500754]
  This paper proposes a new end-to-end text-to-speech (E2E-TTS) model based on neural machine translation (NMT) The proposed model consists of two components; a non-autoregressive vector quantized variational autoencoder (VQ-VAE) model and an autoregressive Transformer-NMT model.
  arXiv  Detail & Related papers  (2020-05-12T02:45:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.