Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories

• URL: http://arxiv.org/abs/2604.25143v1
• Date: Tue, 28 Apr 2026 02:44:43 GMT
• Title: Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories
• Abstract summary: We show that replacing the rolling SVD of AdamW updates with a rolling SVD of loss gradients changes the diagnostic by 1-2 orders of magnitude.<n>Performing SVD on the loss gradient instead of the AdamW update increases the measured perturbative coupling between SED directions and Linear Centroid Hypothesis (LCH) features.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that replacing the rolling SVD of AdamW updates with a rolling SVD of loss gradients changes the diagnostic by 1-2 orders of magnitude. Performing SVD on the loss gradient instead of the AdamW update increases the measured perturbative coupling between SED directions and Linear Centroid Hypothesis (LCH) features from $ \bar{R}_k \approx 3 $--$9\times$ to $100$--$330\times$ across four single-task modular arithmetic operations, eliminating the apparent operation dependence in the original measurement. On a multitask transformer with a shared encoder, update-based SED gives $ \bar{R}_k \leq 1 $ -- an apparent failure of the diagnostic -- while per-operation gradient-based SED recovers $ \bar{R}_k = 20 $--$45\times$ across all four operations. Gradient aggregation across competing tasks is the main obstruction; performing SVD on per-task gradients resolves it. A causal intervention shows that constraining attention updates to any rank-3 subspace (whether SED-derived or random) accelerates grokking by approximately $2.3\times$ across random seeds and operations, while removing the rank-3 component has negligible effect under proper gradient-projection methodology. The SED-LCH coupling is therefore a strong diagnostic of where feature formation concentrates in parameter space, but it is not a unique causal pathway: the natural full-rank AdamW attention update is highly rank-redundant under our hyperparameters.

Related papers

• Gradient-Based Latent Decomposition Reveals Mechanisms of Feature Degradation in Weakly Supervised Mammography [0.0]
  Weakly supervised hierarchical models exhibit a persistent asymmetry.<n> coarse lesion-type features are preserved under reconstruction while fine-grained malignancy cues degrade.<n>In high-dimensional spaces, a single coarse supervisory signal isolates only a 1D latent direction, forcing critical fine-grained features into the vulnerable residual subspace.
  arXiv  Detail & Related papers  (2026-07-24T01:04:07Z)
• The Lie We Tell: Correcting the Euclidean Fallacy in Vision Language Action Policies via Score Matching on Tangent Space [19.950575615573978]
  We introduce a diffusion framework operating intrinsically on SE(3).<n>Our method injects noise through left-invariant SDEs, predicts scores in the space tangent, and retracts samples via the exponential map.<n>On CALVIN ABC$rightarrow$D, LDA improves average task length from $3.27$ to $3.51$ ($+7.3%$)
  arXiv  Detail & Related papers  (2026-06-01T07:59:29Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering [70.99933391739154]
  We focus on two recurring failure modes where the agent repeatedly reasons over information it already has, and where it issues tool calls without integrating recent observations or acquiring new evidence.<n>In this paper, we introduce TACT (Think-Act via activation Steering), to detect and mitigate agent drift in the residual stream before it surfaces as a behavioral failure.<n>In specific, we label trajectory steps as overthinking, overacting, or calibrated, and find that their hidden states can separate linearly along two *drift axes*, pointing from calibrated behavior toward each failure mode.
  arXiv  Detail & Related papers  (2026-05-07T10:24:27Z)
• SA-CycleGAN-2.5D: Self-Attention CycleGAN with Tri-Planar Context for Multi-Site MRI Harmonization [3.5109108807229403]
  We propose SA-CycleGAN-2.5D, a domain adaptation framework motivated by the $HH$-divergence bound of Ben-David et al.<n>By bridging 2D efficiency and 3D consistency, our framework yields voxel-level harmonized images that preserve tumor pathophysiology.
  arXiv  Detail & Related papers  (2026-03-17T23:49:46Z)
• OrthoAI v2: From Single-Agent Segmentation to Dual-Agent Treatment Planning for Clear Aligners [0.0]
  We present OrthoAI v2, the second iteration of our open-source pipeline for AI-assisted orthodontic treatment planning.<n>vtwo addresses all three limitations through three principal contributions.<n>On a synthetic benchmark of 200 crowding scenarios, the parallel ensemble of OrthoAI v2 reaches a planning quality score of $92.8 pm 4.1$ vs. $76.4 pm 8.3$ for OrthoAI v1, a $+21%$ relative gain, while maintaining full CPU deployability ($4.2 pm 0.8$s)
  arXiv  Detail & Related papers  (2026-03-10T13:03:44Z)
• GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models [6.2676602262188625]
  VLA models encode visual observations as 2D patch tokens with no intrinsic geometric structure.<n>We introduce GST-VLA with two contributions.<n>First, the Gaussian Spatial Tokenizer ( GST) converts frozen dense depth and frozen semantic patch features into 3D primitives.<n>Second, 3D Depth-Aware Chain-of-Thought (DA-CoT) reasoning supervises four structured intermediate spatial thoughts.
  arXiv  Detail & Related papers  (2026-03-10T01:39:38Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs [58.19080159470868]
  We propose a novel low-rank ZO estimator, TeZO, which captures the low-rankness across both the model and temporal dimension.<n>Specifically, we represent ZO perturbations along the temporal dimension as a 3D tensor and employ Canonical Polyadic Decomposition (CPD) to extract each low-rank 2D matrix.
  arXiv  Detail & Related papers  (2025-01-31T11:34:03Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Swin UNETR++: Advancing Transformer-Based Dense Dose Prediction Towards Fully Automated Radiation Oncology Treatments [0.0]
  We propose Swin UNETR++, that contains a lightweight 3D Dual Cross-Attention (DCA) module to capture the intra and inter-volume relationships of each patient's anatomy. Our model was trained, validated, and tested on the Open Knowledge-Based Planning dataset.
  arXiv  Detail & Related papers  (2023-11-11T13:52:59Z)
• Training \beta-VAE by Aggregating a Learned Gaussian Posterior with a Decoupled Decoder [0.553073476964056]
  Current practices in VAE training often result in a trade-off between the reconstruction fidelity and the continuity$/$disentanglement of the latent space. We present intuitions and a careful analysis of the antagonistic mechanism of the two losses, and propose a simple yet effective two-stage method for training a VAE. We evaluate the method using a medical dataset intended for 3D skull reconstruction and shape completion, and the results indicate promising generative capabilities of the VAE trained using the proposed method.
  arXiv  Detail & Related papers  (2022-09-29T13:49:57Z)
• Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization [81.29406957201458]
  Existing detection methods commonly use a parameterized bounding box (BBox) to model and detect (horizontal) objects. We argue that such a mechanism has fundamental limitations in building an effective regression loss for rotation detection. We propose to model the rotated objects as Gaussian distributions. We extend our approach from 2-D to 3-D with a tailored algorithm design to handle the heading estimation.
  arXiv  Detail & Related papers  (2022-09-22T07:50:48Z)
• The KFIoU Loss for Rotated Object Detection [115.334070064346]
  In this paper, we argue that one effective alternative is to devise an approximate loss who can achieve trend-level alignment with SkewIoU loss. Specifically, we model the objects as Gaussian distribution and adopt Kalman filter to inherently mimic the mechanism of SkewIoU. The resulting new loss called KFIoU is easier to implement and works better compared with exact SkewIoU.
  arXiv  Detail & Related papers  (2022-01-29T10:54:57Z)
• Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD [15.112499553818953]
  We study the non-asymotic convergence property of the D$2$/Exact-diffusion algorithm. Compared with existing decentralized algorithms, D$2$/Exact-diffusion is least sensitive to network topology.
  arXiv  Detail & Related papers  (2021-05-17T17:16:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.