Sensitivity-Positional Co-Localization in GQA Transformers

• URL: http://arxiv.org/abs/2604.07766v1
• Date: Thu, 09 Apr 2026 03:41:33 GMT
• Title: Sensitivity-Positional Co-Localization in GQA Transformers
• Abstract summary: We investigate a fundamental structural question in Grouped Query Attention transformers.<n>Do the layers most sensitive to task correctness coincide with the layers where positional encoding adaptation has the greatest leverage?<n>We introduce LSLORA, which restricts LoRA adaptation to layers identified via a novel correctness-differential hidden-state metric, and GARFA, which attaches 8 learnable per-KV-head scalar multipliers to each targeted layer.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate a fundamental structural question in Grouped Query Attention (GQA) transformers: do the layers most sensitive to task correctness coincide with the layers where positional encoding adaptation has the greatest leverage? We term this the co-localization hypothesis and test it on Llama 3.1 8B, a 32-layer GQA model with a 4:1 query-to-key-value head ratio. We introduce \LSLORA, which restricts LoRA adaptation to layers identified via a novel correctness-differential hidden-state metric, and GARFA (GQA-Aware RoPE Frequency Adaptation), which attaches 8 learnable per-KV-head scalar multipliers to each targeted layer. Contrary to the co-localization hypothesis, we discover strong anti-localization: task-sensitive layers concentrate in the late network ($\ell\in\{23\text{-}31\}$) while RoPE-influential layers dominate the early network ($\ell\in\{0\text{-}9\}$), yielding Spearman $r_s = -0.735$ ($p = 1.66\times10^{-6}$). Despite this anti-localization, a 4-way cross-layer ablation shows that applying both interventions to the sensitivity-identified layers outperforms all alternative configurations by 4-16 percentage points across six diverse benchmarks (MMLU, GPQA, HumanEval+, MATH, MGSM, ARC), approaching Claude 3.5 Haiku on HumanEval+ (67.1% vs. 68.3%) at \$100 total compute cost.

Related papers

• Can Scene Text Recognition Read Rare Compositions? [1.7844382164707184]
  Scene text recognition is reported as 89--97% accurate on the six standard benchmarks, and the problem is widely treated as saturated.<n>We present an alternative reading.<n>When the same test images are stratified jointly by ground-truth word rarity and character n-gram novelty against a reference corpus, accuracy at the rareword x rare-trigram corner of the resulting 5x5 grid drops 10--18 pt below the q3/q3 centre.
  arXiv  Detail & Related papers  (2026-09-01T07:16:42Z)
• Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation [18.8104643146881]
  We study the choice of constructor-efficient fine-tuning under a severe trainable-state budget.<n>We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates.<n>On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks.
  arXiv  Detail & Related papers  (2026-09-01T05:52:38Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Rethinking Attention Locality in Spiking Transformers [50.00501275379363]
  Spiking Transformers provide a promising paradigm for efficient visual processing with spike-driven computation.<n>Their Softmax-free Spiking Self-Attention (SSA) struggles to establish spatially localized token interactions.<n>We propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP)<n>Our approach improves mAP@50 by up to 9.50% on COCO on 2017 and mIoU by up to 3.42% on ADE20K.
  arXiv  Detail & Related papers  (2026-08-09T07:36:17Z)
• The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning [3.860898721527665]
  LoRA fine-tuning can create intruder dimensions that drive catastrophic forgetting.<n>No theory has predicted, layer by layer on measured spectra, when they appear.<n>Full fine-tuning disperses its update far below the threshold of every layer.<n>Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting.
  arXiv  Detail & Related papers  (2026-07-26T15:17:41Z)
• TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging [6.356259006504384]
  We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry across transformer architectures.<n>We propose a training-free algorithm that exploits this structure through calibrated probe-guided per-layer gating and per-component subspace-aware merging.<n>$textbfTaDA$ produces a standard rank-$r$ LoRA adapter with zero inference overhead.
  arXiv  Detail & Related papers  (2026-06-03T15:39:37Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution [11.642796814179926]
  We systematize learning rate scheduling into five generations: (Gen1) global fixed learning rates, (Gen2) global scheduling, (Gen3) parameter-level adaptation, (Gen4) layer-level differentiation, and (Gen5) joint layer-time scheduling.<n>We propose Discriminative Adaptive Layer Scaling (DALS), a unified framework that integrates phase-adaptive cosine scheduling, depth-aware Grokfast filtering, and LARS-style trust ratios into a single coherent gradient.<n>We benchmark 18 strategies including three DALS variants across all five generations on five datasets: synthetic, CIFAR-10 (from scratch), RTE
  arXiv  Detail & Related papers  (2026-04-30T01:13:48Z)
• Partial Number Theoretic Transform Masking in Post Quantum Cryptography Hardware: A Security Margin Analysis [0.0]
  Adams Bridge, a hardware accelerator for ML-DSA and ML-KEM, masks 1 of its Inverse Number Theoretic Transform layers and relies on shuffling for the remainder.<n>We evaluate these claims against published side-channel literature across seven analysis tracks with confidence-rated evidence.
  arXiv  Detail & Related papers  (2026-04-04T17:38:34Z)
• Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations [2.7946918847372277]
  Large language models demonstrate strong performance on mathematical reasoning benchmarks, yet remain surprisingly fragile to meaning-preserving surface perturbations.<n>We systematically evaluate three open-weight LLMs, Mistral-7B, Llama-3-8B, and Qwen2.5-7B, on 677 GSM8K problems paired with semantically equivalent variants.<n>All three models exhibit substantial answer-flip rates (28.8%-45.1%), with number paraphrasing consistently more disruptive than name swaps.
  arXiv  Detail & Related papers  (2026-04-02T05:30:20Z)
• LAAF: Logic-layer Automated Attack Framework A Systematic Red-Teaming Methodology for LPCI Vulnerabilities in Agentic Large Language Model Systems [0.39875976220956705]
  LAAF is the first automated red-teaming framework to combine an LPCI-specific technique taxonomy with stage-sequential seed escalation.<n>We show that LAAF achieves higher stage-breakthrough efficiency than single-technique random testing.
  arXiv  Detail & Related papers  (2026-03-18T00:51:36Z)
• Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks [0.0]
  We present a practical system for privacy-aware large language model (LLM) inference that splits a transformer between a trusted local GPU and a cloud GPU.<n>Our system addresses the unique challenges of autoregressive LLM decoding over high-latency wide-area networks (WANs)<n>System generalizes to larger models with only 4.9 GB local VRAM and matching 7B throughput.
  arXiv  Detail & Related papers  (2026-02-18T14:13:08Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Void in Language Models [2.7195102129095003]
  We analyze layer activation in instruction-tuned language models (LMs) across two phases: Prompt Processing and Response Generation.<n>We show that not all layers contribute equally during inference, and that selectively skipping most of them can improve the performance of models on certain tasks.
  arXiv  Detail & Related papers  (2025-05-20T15:01:56Z)
• Cross-layer Attention Sharing for Pre-trained Large Language Models [87.23702516022435]
  We introduce LISA, a lightweight substitute for self-attention in large language models (LLMs)<n>We show that LISA maintains high response quality in terms of accuracy and perplexity while reducing redundant attention calculations within 53%-84% of the total layers.<n>Our implementations of LISA achieve a 6x compression of Q and K matrices within the attention mechanism, with maximum throughput improvements 19.5%, 32.3%, and 40.1% for LLaMA3-8B, LLaMA2-7B, and LLaMA2-13B, respectively.
  arXiv  Detail & Related papers  (2024-08-04T00:38:34Z)
• ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention [48.697458429460184]
  Two factors, information bottleneck sensitivity and inconsistency between different attention topologies, could affect the performance of the Sparse Transformer. This paper proposes a well-designed model named ERNIE-Sparse. It consists of two distinctive parts: (i) Hierarchical Sparse Transformer (HST) to sequentially unify local and global information, and (ii) Self-Attention Regularization (SAR) to minimize the distance for transformers with different attention topologies.
  arXiv  Detail & Related papers  (2022-03-23T08:47:01Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.