ZoDIAC: Zoneout Dropout Injection Attention Calculation

• URL: http://arxiv.org/abs/2206.14263v4
• Date: Tue, 30 Sep 2025 22:40:17 GMT
• Title: ZoDIAC: Zoneout Dropout Injection Attention Calculation
• Abstract summary: We introduce a novel refine and intensify attention mechanism called Zoneup Dropout Injection Attention Calculation (ZoDIAC)<n>Our experiments show that ZoDIAC achieves statistically significant higher scores under all image captioning metrics.<n>Our proposed ZoDIAC attention modules can be used as a drop-in replacement for the attention components in all transformer models.
• Score: 5.027714423258536
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In the past few years the transformer model has been utilized for a variety of tasks such as image captioning, image classification natural language generation, and natural language understanding. As a key component of the transformer model, self-attention calculates the attention values by mapping the relationships among the head elements of the source and target sequence, yet there is no explicit mechanism to refine and intensify the attention values with respect to the context of the input and target sequences. Based on this intuition, we introduce a novel refine and intensify attention mechanism that is called Zoneup Dropout Injection Attention Calculation (ZoDIAC), in which the intensities of attention values in the elements of the input source and target sequences are first refined using GELU and dropout and then intensified using a proposed zoneup process which includes the injection of a learned scalar factor. Our extensive experiments show that ZoDIAC achieves statistically significant higher scores under all image captioning metrics using various feature extractors in comparison to the conventional self-attention module in the transformer model on the MS-COCO dataset. Our proposed ZoDIAC attention modules can be used as a drop-in replacement for the attention components in all transformer models. The code for our experiments is publicly available at: https://github.com/zanyarz/zodiac

Related papers

• Bottleneck Transformer-Based Approach for Improved Automatic STOI Score Prediction [16.426476430697587]
  We present a novel approach to predict the Short-Time Objective Intelligibility (STOI) metric using a bottleneck transformer architecture.<n>Our model has shown higher correlation and lower mean squared error for both seen and unseen scenarios.
  arXiv  Detail & Related papers  (2026-02-17T10:46:54Z)
• Untwisting RoPE: Frequency Control for Shared Attention in DiTs [84.14005261938284]
  Positional encodings are essential to transformer-based generative models.<n>We show that Rotary Positional Embeddings (RoPE) naturally decomposes into frequency components with distinct positional sensitivities.<n>We introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment.
  arXiv  Detail & Related papers  (2026-02-04T20:01:59Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• Rectifying Magnitude Neglect in Linear Attention [57.097694292570885]
  Linear Attention suffers from a significant performance degradation compared to standard Softmax Attention.<n>We propose Magnitude-Aware Linear Attention (MALA), which modifies the computation of Linear Attention to fully incorporate the Query's magnitude.
  arXiv  Detail & Related papers  (2025-07-01T11:49:05Z)
• Focus What Matters: Matchability-Based Reweighting for Local Feature Matching [6.361840891399624]
  We propose a novel attention reweighting mechanism that simultaneously incorporates a learnable bias term into the attention logits.<n>Experiments conducted on three benchmark datasets validate the effectiveness of our method.
  arXiv  Detail & Related papers  (2025-05-04T15:50:28Z)
• Learning to Attribute with Attention [75.61481181755744]
  We propose treating attention weights of different attention heads as features. This way, we can learn how to effectively leverage attention weights for attribution. Our resulting method, Attribution with Attention (AT2), reliably performs on par with approaches that involve many ablations.
  arXiv  Detail & Related papers  (2025-04-18T15:36:28Z)
• Neural Attention: A Novel Mechanism for Enhanced Expressive Power in Transformer Models [0.0]
  Transformer models typically calculate attention matrices using dot products.<n>We propose Neural Attention, a technique that replaces dot products with feed-forward networks.<n>This approach modifies only the attention matrix calculation while preserving the matrix dimensions.
  arXiv  Detail & Related papers  (2025-02-24T14:39:40Z)
• Core Context Aware Attention for Long Context Language Modeling [50.774702091154204]
  We propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-range context modeling. Our CCA-Attention significantly outperforms state-of-the-art models in terms of computational efficiency and long-context modeling ability.
  arXiv  Detail & Related papers  (2024-12-17T01:54:08Z)
• Continual Low-Rank Scaled Dot-product Attention [67.11704350478475]
  We introduce a new formulation of the Scaled-product Attention based on the Nystr"om approximation that is suitable for Continual Inference.<n>In experiments on Online Audio Classification and Online Action Detection tasks, the proposed Continual Scaled Dot-product Attention can lower the number of operations by up to three orders of magnitude.
  arXiv  Detail & Related papers  (2024-12-04T11:05:01Z)
• DAPE V2: Process Attention Score as Feature Map for Length Extrapolation [63.87956583202729]
  We conceptualize attention as a feature map and apply the convolution operator to mimic the processing methods in computer vision. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution.
  arXiv  Detail & Related papers  (2024-10-07T07:21:49Z)
• Continuous Sign Language Recognition Using Intra-inter Gloss Attention [0.0]
  In this study, we introduce a novel module in sign language recognition studies, called intra-inter gloss attention module. In the intra-gloss attention module, the video is divided into equally sized chunks and a self-attention mechanism is applied within each chunk. Experimental results on the PHOENIX-2014 benchmark dataset demonstrate that our method can effectively extract sign language features in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-06-26T13:21:08Z)
• Elliptical Attention [1.7597562616011944]
  Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. We propose using a Mahalanobis distance metric for computing the attention weights to stretch the underlying feature space in directions of high contextual relevance.
  arXiv  Detail & Related papers  (2024-06-19T18:38:11Z)
• Prophet Attention: Predicting Attention with Future Attention for Image Captioning [99.88870695151874]
  We propose the Prophet Attention, similar to the form of self-supervision. The proposed Prophet Attention can be easily incorporated into existing image captioning models.
  arXiv  Detail & Related papers  (2022-10-19T22:29:31Z)
• Hybrid Routing Transformer for Zero-Shot Learning [83.64532548391]
  This paper presents a novel transformer encoder-decoder model, called hybrid routing transformer (HRT) We embed an active attention, which is constructed by both the bottom-up and the top-down dynamic routing pathways to generate the attribute-aligned visual feature. While in HRT decoder, we use static routing to calculate the correlation among the attribute-aligned visual features, the corresponding attribute semantics, and the class attribute vectors to generate the final class label predictions.
  arXiv  Detail & Related papers  (2022-03-29T07:55:08Z)
• Vision Transformer with Deformable Attention [29.935891419574602]
  Large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. We propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. We present Deformable Attention Transformer, a general backbone model with deformable attention for both image classification and dense prediction tasks.
  arXiv  Detail & Related papers  (2022-01-03T08:29:01Z)
• Causal Attention for Unbiased Visual Recognition [76.87114090435618]
  Attention module does not always help deep models learn causal features that are robust in any confounding context. We propose causal attention module (CaaM) that self-annotates the confounders in unsupervised fashion. In OOD settings, deep models with CaaM outperform those without it significantly.
  arXiv  Detail & Related papers  (2021-08-19T16:45:51Z)
• Combiner: Full Attention Transformer with Sparse Computation Cost [142.10203598824964]
  We propose Combiner, which provides full attention capability in each attention head while maintaining low computation complexity. We show that most sparse attention patterns used in existing sparse transformers are able to inspire the design of such factorization for full attention. An experimental evaluation on both autoregressive and bidirectional sequence tasks demonstrates the effectiveness of this approach.
  arXiv  Detail & Related papers  (2021-07-12T22:43:11Z)
• Capturing Multi-Resolution Context by Dilated Self-Attention [58.69803243323346]
  We propose a combination of restricted self-attention and a dilation mechanism, which we refer to as dilated self-attention. The restricted self-attention allows attention to neighboring frames of the query at a high resolution, and the dilation mechanism summarizes distant information to allow attending to it with a lower resolution. ASR results demonstrate substantial improvements compared to restricted self-attention alone, achieving similar results compared to full-sequence based self-attention with a fraction of the computational costs.
  arXiv  Detail & Related papers  (2021-04-07T02:04:18Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Centroid Transformers: Learning to Abstract with Attention [15.506293166377182]
  Self-attention is a powerful mechanism for extracting features from the inputs. We propose centroid attention, a generalization of self-attention that maps N inputs to M outputs $(Mleq N)$. We apply our method to various applications, including abstractive text summarization, 3D vision, and image processing.
  arXiv  Detail & Related papers  (2021-02-17T07:04:19Z)
• Text Information Aggregation with Centrality Attention [86.91922440508576]
  We propose a new way of obtaining aggregation weights, called eigen-centrality self-attention. We build a fully-connected graph for all the words in a sentence, then compute the eigen-centrality as the attention score of each word.
  arXiv  Detail & Related papers  (2020-11-16T13:08:48Z)
• Boost Image Captioning with Knowledge Reasoning [10.733743535624509]
  We propose word attention to improve the correctness of visual attention when generating sequential descriptions word-by-word. We introduce a new strategy to inject external knowledge extracted from knowledge graph into the encoder-decoder framework to facilitate meaningful captioning.
  arXiv  Detail & Related papers  (2020-11-02T12:19:46Z)
• Structured Self-Attention Weights Encode Semantics in Sentiment Analysis [13.474141732019099]
  We show that self-attention scores encode semantics by considering sentiment analysis tasks. We propose a simple and effective Attention Tracing (LAT) method to analyze structured attention weights. Our results show that structured attention weights encode rich semantics in sentiment analysis, and match human interpretations of semantics.
  arXiv  Detail & Related papers  (2020-10-10T06:49:25Z)
• Hard Non-Monotonic Attention for Character-Level Transduction [65.17388794270694]
  We introduce an exact, exponential-time algorithm for marginalizing over a number of non-monotonic alignments between two strings. We compare soft and hard non-monotonic attention experimentally and find that the exact algorithm significantly improves performance over the approximation and outperforms soft attention.
  arXiv  Detail & Related papers  (2018-08-29T20:00:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.