Supervised Attention in Sequence-to-Sequence Models for Speech Recognition

• URL: http://arxiv.org/abs/2204.12308v1
• Date: Mon, 25 Apr 2022 15:38:48 GMT
• Title: Supervised Attention in Sequence-to-Sequence Models for Speech Recognition
• Abstract summary: We show that attention weights produced by models trained end to end do not always correspond well with actual alignments. We treat the correspondence between attention weights and alignments as a learning problem by imposing a supervised attention loss. Experiments have shown significant improved performance, suggesting that learning the alignments well during training critically determines the performance of sequence-to-sequence models.
• Score: 15.44609280660252
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Attention mechanism in sequence-to-sequence models is designed to model the alignments between acoustic features and output tokens in speech recognition. However, attention weights produced by models trained end to end do not always correspond well with actual alignments, and several studies have further argued that attention weights might not even correspond well with the relevance attribution of frames. Regardless, visual similarity between attention weights and alignments is widely used during training as an indicator of the models quality. In this paper, we treat the correspondence between attention weights and alignments as a learning problem by imposing a supervised attention loss. Experiments have shown significant improved performance, suggesting that learning the alignments well during training critically determines the performance of sequence-to-sequence models.

Related papers

• Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners [33.10721304996431]
  Self-supervised learning has driven substantial progress in audio representation learning.<n>We introduce NAPE, a self-supervised framework in which a causal Transformer predicts each next patch embedding of a spectrogram from the previous ones.<n>It achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results.
  arXiv  Detail & Related papers  (2026-08-20T10:16:07Z)
• Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models [129.8880969196575]
  We present an empirical analysis of attention in Diffusion large language models (dLLMs)<n>We show that tokens attending more strongly to unmasked context exhibit greater generation stability and play a critical role in reasoning.<n>Motivated by these findings, we propose AGDO, an attention-guided denoising and optimization framework.
  arXiv  Detail & Related papers  (2026-06-10T16:14:23Z)
• Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention [0.21485350418225238]
  We propose Boltzmann attention, an energy-based generalization in which attention patterns are governed by an interacting Ising model.<n>Experiments on character-level language modeling and synthetic bracket matching show that Boltzmann attention consistently improves over standard softmax attention.
  arXiv  Detail & Related papers  (2026-06-10T06:58:49Z)
• FAIR: Focused Attention Is All You Need for Generative Recommendation [43.65370600297507]
  We propose the first generative recommendation framework with focused attention, which enhances attention scores to relevant context while suppressing those to irrelevant ones.<n>Specifically, we propose (1) a focused attention mechanism integrated into the standard Transformer, which learns two separate sets of Q and K attention weights and computes their difference as the final attention scores.<n>We validate the effectiveness of FAIR on four public benchmarks, demonstrating its superior performance compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T03:25:12Z)
• Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation [62.14692332209628]
  "Interaction Distillation" is a novel training framework for more adequate preference modeling through attention-level optimization.<n>It provides more stable and generalizable reward signals compared to state-of-the-art RM optimization methods.
  arXiv  Detail & Related papers  (2025-08-04T17:06:23Z)
• Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration [1.7373859011890633]
  Large vision-language models (LVLMs) achieve impressive performance on multimodal tasks but often suffer from hallucination.<n>We introduce the Confidence-Aware Attention framework to address this challenge.
  arXiv  Detail & Related papers  (2025-05-27T17:45:21Z)
• AttentionPredictor: Temporal Pattern Matters for Efficient LLM Inference [51.1972443343829]
  We propose AttentionPredictor, which is the first learning-based critical token identification approach.<n> AttentionPredictor accurately predicts the attention score while consuming negligible memory.<n>We also propose a cross-token critical cache prefetching framework that hides the token time overhead to accelerate the decoding stage.
  arXiv  Detail & Related papers  (2025-02-06T13:41:46Z)
• Prophet Attention: Predicting Attention with Future Attention for Image Captioning [99.88870695151874]
  We propose the Prophet Attention, similar to the form of self-supervision. The proposed Prophet Attention can be easily incorporated into existing image captioning models.
  arXiv  Detail & Related papers  (2022-10-19T22:29:31Z)
• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition [4.14099371030604]
  We present our submission to the 3rd Affective Behavior Analysis in-the-wild (ABAW) challenge. Recurrence and attention are the two widely used sequence modelling mechanisms in the literature. We show that LSTM-RNNs can outperform the attention models when coupled with low-complex CNN backbones.
  arXiv  Detail & Related papers  (2022-03-24T18:22:56Z)
• Alignment Attention by Matching Key and Query Distributions [48.93793773929006]
  This paper introduces alignment attention that explicitly encourages self-attention to match the distributions of the key and query within each head. It is simple to convert any models with self-attention, including pre-trained ones, to the proposed alignment attention. On a variety of language understanding tasks, we show the effectiveness of our method in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-10-25T00:54:57Z)
• Impact of Attention on Adversarial Robustness of Image Classification Models [0.9176056742068814]
  Adrial attacks against deep learning models have gained significant attention. Recent works have proposed explanations for the existence of adversarial examples and techniques to defend the models against these attacks. This work aims at a general understanding of the impact of attention on adversarial robustness.
  arXiv  Detail & Related papers  (2021-09-02T13:26:32Z)
• More Than Just Attention: Learning Cross-Modal Attentions with Contrastive Constraints [63.08768589044052]
  We propose Contrastive Content Re-sourcing ( CCR) and Contrastive Content Swapping ( CCS) constraints to address such limitation. CCR and CCS constraints supervise the training of attention models in a contrastive learning manner without requiring explicit attention annotations. Experiments on both Flickr30k and MS-COCO datasets demonstrate that integrating these attention constraints into two state-of-the-art attention-based models improves the model performance.
  arXiv  Detail & Related papers  (2021-05-20T08:48:10Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models [5.866941279460248]
  We propose a visual analytics approach to understanding fine-tuning in attention-based language models. Our visualization, Attention Flows, is designed to support users in querying, tracing, and comparing attention within layers, across layers, and amongst attention heads in Transformer-based language models.
  arXiv  Detail & Related papers  (2020-09-03T19:56:30Z)
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers [55.40032342541187]
  We pre-train a transformer-based model with attention algorithms in a self-supervised fashion and treat them as feature extractors on downstream tasks. Our approach shows comparable performance to the typical self-attention yet requires 20% less time in both training and inference.
  arXiv  Detail & Related papers  (2020-06-09T10:40:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.