Explicit Dropout: Deterministic Regularization for Transformer Architectures

• URL: http://arxiv.org/abs/2604.20505v1
• Date: Wed, 22 Apr 2026 12:45:51 GMT
• Title: Explicit Dropout: Deterministic Regularization for Transformer Architectures
• Abstract summary: Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through masking.<n>We propose a deterministic formulation that expresses dropout as an additive regularizer directly incorporated into the training loss.
• Score: 55.09895958546215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through stochastic masking rather than explicit optimization objectives. We propose a deterministic formulation that expresses dropout as an additive regularizer directly incorporated into the training loss. The framework derives explicit regularization terms for Transformer architectures, covering attention query, key, value, and feed-forward components with independently controllable strengths. This formulation removes reliance on stochastic perturbations while providing clearer and fine-grained control over regularization strength. Experiments across image classification, temporal action detection, and audio classification show that explicit dropout matches or outperforms conventional implicit methods, with consistent gains when applied to attention and feed-forward network layers. Ablation studies demonstrate stable performance and controllable regularization through regularization coefficients and dropout rates. Overall, explicit dropout offers a practical and interpretable alternative to stochastic regularization while maintaining architectural flexibility across diverse tasks.

Related papers

• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• Improving Adversarial Robustness of Attribution via Implicit Regularization [9.78455276849371]
  adversarial robustness of attributions is a fundamental requirement for reliable explainability in deep learning.<n>We show that robustness can arise implicitly from the learning dynamics of standard gradient descent.<n>Our results highlight learning dynamics as a principled and practical mechanism for robust explainability.
  arXiv  Detail & Related papers  (2026-05-28T14:19:39Z)
• REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning [5.853890728403084]
  Recent text models are often adapted to specialized domains via contrastive pre-finetuning (PFT) on a collection of scattered, heterogeneous tasks.<n>This approach often introduces task-induced bias alongside domain knowledge, leading to uncontrolled representation shifts.<n>We propose R, a representation regularization framework that explicitly controls representation shift during embedding pre-finetuning.
  arXiv  Detail & Related papers  (2026-04-19T04:41:55Z)
• Likelihood-guided Regularization in Attention Based Models [1.561268797057701]
  We propose a likelihood-guided variational Ising-based regularization framework for Vision Transformers (ViTs)<n>We show that the Ising regularizer leads to better-calibrated probability estimates and structured feature selection through uncertainty-aware attention mechanisms.
  arXiv  Detail & Related papers  (2025-11-17T10:38:09Z)
• AttentionDrop: A Novel Regularization Method for Transformer Models [0.3262230127283452]
  Transformer-based architectures achieve state-of-the-art performance across a wide range of tasks in natural language processing, computer vision, and speech processing.<n>However, their immense capacity often leads to overfitting, especially when training data is limited or noisy.<n>This research proposes a unified family of regularization techniques, which operate directly on the self-attention distributions.
  arXiv  Detail & Related papers  (2025-04-16T13:51:16Z)
• Structured Context Recomposition for Large Language Models Using Probabilistic Layer Realignment [0.0]
  This paper introduces a probabilistic layer realignment strategy that dynamically adjusts learned representations within transformer layers.<n>It mitigates abrupt topic shifts and logical inconsistencies, particularly in scenarios where sequences exceed standard attention window constraints.<n>While SCR incurs a moderate increase in processing time, memory overhead remains within feasible limits, making it suitable for practical deployment in autoregressive generative applications.
  arXiv  Detail & Related papers  (2025-01-29T12:46:42Z)
• Regularization for Adversarial Robust Learning [18.46110328123008]
  We develop a novel approach to adversarial training that integrates $phi$-divergence regularization into the distributionally robust risk function. This regularization brings a notable improvement in computation compared with the original formulation. We validate our proposed method in supervised learning, reinforcement learning, and contextual learning and showcase its state-of-the-art performance against various adversarial attacks.
  arXiv  Detail & Related papers  (2024-08-19T03:15:41Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Normalization and effective learning rates in reinforcement learning [52.59508428613934]
  Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature. We show that normalization brings with it a subtle but important side effect: an equivalence between growth in the norm of the network parameters and decay in the effective learning rate. We propose to make the learning rate schedule explicit with a simple re- parameterization which we call Normalize-and-Project.
  arXiv  Detail & Related papers  (2024-07-01T20:58:01Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Causally-motivated Shortcut Removal Using Auxiliary Labels [63.686580185674195]
  Key challenge to learning such risk-invariant predictors is shortcut learning. We propose a flexible, causally-motivated approach to address this challenge. We show both theoretically and empirically that this causally-motivated regularization scheme yields robust predictors.
  arXiv  Detail & Related papers  (2021-05-13T16:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.