Attention Sinks and Outliers in Attention Residuals

• URL: http://arxiv.org/abs/2605.17887v1
• Date: Mon, 18 May 2026 05:53:09 GMT
• Title: Attention Sinks and Outliers in Attention Residuals
• Abstract summary: We propose an outlier- and sink-aware technique built on inter-layer null signaling.<n>AttnResidual architectures introduce an additional depth-wise normalization channel.<n>We show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness.
• Score: 39.45624457227565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4.

Related papers

• WaVeFuse: Regime-Adaptive Equity Index Forecasting via Channel-Wise Wavelet Denoising and Vertical Attention Fusion [0.0]
  WaVeFuse is a hybrid Deep Learning for equity forecasting framework.<n>It addresses propagation of technical indicators (TIs), channel-indiscriminate decomposition, and static multi-branch fusion.<n>WaVeFuse delivers a computationally efficient, regime-robust framework suitable for research and decision-support deployment.
  arXiv  Detail & Related papers  (2026-09-13T18:50:33Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Structure-Enhanced Features and Quality-Aware Dynamic Anchor Scoring for Robust Lane Detection [3.276955471607996]
  Lane detection requires recovering thin, elongated, and frequently occluded lane structures under challenging driving conditions.<n>We propose a structure-enhanced and quality-aware framework that improves lane representation and dynamic-anchor scoring.
  arXiv  Detail & Related papers  (2026-08-10T13:52:04Z)
• The Rescue Effect: Spatio-Semantic Early Exit Bypasses Quantization Collapse in CLIP [6.450516062706025]
  In joint-embedding architectures such as CLIP this introduces a failure mode distinct from quantized CNN classifiers.<n>We characterize this as Quantization-Induced Representation Collapse (QIRC) and quantify it on INT8 replacing CLIP ViT-B/32.<n>We propose LRA-EE (Layer-wise Representation-Aware Early Exit), which bypasses noise-saturated deep layers.<n>On ImageNet-1K zero-shot classification, LRA-EE reduces FLOPs by 13.4% and improves Top-1 accuracy by +2.44%p (58.72% -> 61.16%) over the INT8 baseline
  arXiv  Detail & Related papers  (2026-05-26T00:45:46Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension [5.50723299083202]
  We propose OSC, a hardware-efficient framework for outlier suppression.<n> OSC is highly hardware-friendly, achieving a peak speedup of 1.78x over the W8A8 GEMM baseline on a modern AI accelerator.
  arXiv  Detail & Related papers  (2026-04-14T14:17:59Z)
• OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance [53.45125687428705]
  Open-Vocabulary aims to segment image regions beyond predefined category sets by leveraging semantic descriptions.<n>Recent efforts have incorporated Vision Foundation Models (VFMs) like DINO to alleviate these limitations.<n>We propose OVS-DINO, a novel framework that revitalizes latent edge-sensitivity of DINO through structural alignment.
  arXiv  Detail & Related papers  (2026-04-09T16:57:11Z)
• Distributional Reinforcement Learning with Information Bottleneck for Uncertainty-Aware DRAM Equalization [8.695939803795499]
  We propose a distributional risk-sensitive reinforcement learning framework integrating Information Bottleneck latent representations with Conditional Value-at-Risk optimization.<n>We introduce rate-distortion optimal signal compression achieving 51 times speedup over eye diagrams.<n>We show that the proposed framework provides a practical solution for production-scale equalizer optimization with certified worst-case guarantees.
  arXiv  Detail & Related papers  (2026-03-05T03:34:25Z)
• Adaptive Enhancement and Dual-Pooling Sequential Attention for Lightweight Underwater Object Detection with YOLOv10 [0.0]
  This manuscript introduces a streamlined yet robust framework for underwater object detection, grounded in the YOLOv10 architecture.<n>The proposed method integrates a Multi-Stage Adaptive Enhancement module to improve image quality and a Dual-Pooling Sequential Attention mechanism to strengthen multi-scale feature representation.
  arXiv  Detail & Related papers  (2026-03-04T07:39:57Z)
• Blockchain-Enabled Routing for Zero-Trust Low-Altitude Intelligent Networks [77.17664010626726]
  We focus on the routing with multiple UAV clusters in low-altitude intelligent networks (LAINs)<n>To minimize the damage caused by potential threats, we present the zero-trust architecture with the software-defined perimeter and blockchain techniques.<n>We show that the proposed framework reduces the average E2E delay by 59% and improves the TSR by 29% on average compared to benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T04:30:35Z)
• N-EIoU-YOLOv9: A Signal-Aware Bounding Box Regression Loss for Lightweight Mobile Detection of Rice Leaf Diseases [0.6280530476948474]
  We propose N EIoU YOLOv9, a lightweight detection framework based on a signal aware bounding box regression loss.<n>The proposed loss reshapes localization gradient by combining non monotonic focusing with decoupled width and height optimization.<n>This design is particularly effective for small and low contrast targets commonly observed in agricultural disease imagery.
  arXiv  Detail & Related papers  (2026-01-14T05:13:36Z)
• DARN: Dynamic Adaptive Regularization Networks for Efficient and Robust Foundation Model Adaptation [0.0]
  We introduce Dynamic Adaptive Regularization Networks (DARN)<n>DARN integrates three key innovations: a lightweight Task Complexity Predictor ( TCP) that estimates per-sample difficulty, Adaptive Dropout Modulation (ADM) and Dynamic Capacity Gating (DCG)<n>In full fine-tuning (unfrozen backbone), DARN achieves a new state-of-the-art on the multi-task GeoBench benchmark (86.66% mIoU, +5.56 pp over prior SOTA).<n>In efficient adaptation (frozen backbone), DARN achieves SOTA-competitive accuracy (90.5% mIoU on Sen
  arXiv  Detail & Related papers  (2025-11-06T19:36:49Z)
• Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning [73.10669391954801]
  We present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0.<n>Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention.<n>Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%.
  arXiv  Detail & Related papers  (2025-10-22T07:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.