Distributional Spectral Diagnostics for Localizing Grokking Transitions

• URL: http://arxiv.org/abs/2605.08237v1
• Date: Thu, 07 May 2026 03:08:58 GMT
• Title: Distributional Spectral Diagnostics for Localizing Grokking Transitions
• Abstract summary: In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize.<n>We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off.
• Score: 4.480801209867367
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize. We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off. Task-dependent observables are summarized as empirical distributions, mapped to Wasserstein/quantile coordinates, and analyzed by Hankel dynamic mode decomposition (DMD); the resulting reconstruction residual, together with spectrum and effective rank, forms the diagnostic output. On held-out modular-addition Transformer runs, the residual achieves AUROC \(\approx \) 0.93 for grokking-vs-non-grokking discrimination at the run level; under a fixed sustained-threshold operating rule, true-positive alarms can precede onset, with lead time reported jointly with false-alarm rate and uncertainty intervals. Perturbation experiments show that, in the tested \(wd=1\) pool, high-residual windows exhibit about \(3\times\) larger short-horizon perturbation deviation than low-residual windows. In a same-data norm-window control, perturbation sensitivity aligns with the residual ordering rather than total-parameter-norm ordering, suggesting that the residual is not merely a total-norm proxy at the window level in the studied \(wd=1\) dynamics. Norm signals remain strong run-level regime indicators, and log-probability performs best among the observables tested under the current protocol. We position the residual as a window-level monitoring and localization signal in the studied modular-arithmetic Transformer settings, not a universal early-warning predictor or an intervention rule.

Related papers

• Change Detection in Probability Flow ODE: Online Testing in Diffusion Latent Spaces [0.0]
  A growing range of sequential data tasks, such as identifying trend reversals in financial markets, cannot be fully addressed without detection of distributional shifts in time-ordered data.<n>We consider a sequential change-point detection problem where the conditional density switches at an unknown time, yet neither the pre- nor post-change distribution admits a closed-form.<n>A conditional diffusion model, trained on pre-change-point data with a frozen context encoder, defines a deterministicection via the probability flow ODE.
  arXiv  Detail & Related papers  (2026-08-24T05:09:24Z)
• Actionable Hallucination Detection: Translating Latent Uncertainty into Agentic Critique [1.1172382217477128]
  Large Language Models (LLMs) deployed as AI agents frequently exhibit user specification-grounding failures.<n>Existing detection methods fail to provide actionable, real-time correction as they either do not localize the hallucinations, or incur prohibitive inference latency.<n>We introduce the Latent Critic, a lightweight low-rank adapter (LoRA) that operates concurrently with a frozen base LLM's generation.
  arXiv  Detail & Related papers  (2026-08-11T03:26:16Z)
• Graph Diffusion Residuals for Control-Function Instrumental Variables [82.39671985893028]
  High-capacity first stages can interpolate treatment and leave too little residual information for the outcome equation.<n>We study Adaptive Anisotropic Instrumental Heat Flow (A-IHF), a deterministic graph-diffusion residual extractor for flexible control functions.<n>A-IHF treats treatment as a signal on a graph of first-stage features, uses pilot diffusion to detect large treatment jumps, attenuates conductance across those jumps, and computes the generated control with a sparse graph resolvent.
  arXiv  Detail & Related papers  (2026-06-12T17:02:53Z)
• Kurtosis-Guided Denoising Score Matching for Tabular Anomaly Detection [6.893608092072852]
  Denoising score matching (DSM) provides a way to learn data distributions by training a neural network to recover the score function.<n>We introduce kurtosis-based noise scaling (K-DSM), a per-feature scheme that sets noise levels from the shape of each marginal distribution.<n>On standard anomaly detection benchmarks, K-DSM achieves state-of-the-art performance in the semi-supervised setting.
  arXiv  Detail & Related papers  (2026-05-07T21:19:12Z)
• The Stationarity Bias: Stratified Stress-Testing for Time-Series Imputation in Regulated Dynamical Systems [0.098314893665023]
  Time-series imputation benchmarks use random masking and shape-agnostic metrics.<n>We formalize this bias and propose a emphStratified Stress-Test that partitions evaluation into Stationary and Transient regimes.
  arXiv  Detail & Related papers  (2026-02-17T15:05:56Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Correcting False Alarms from Unseen: Adapting Graph Anomaly Detectors at Test Time [60.341117019125214]
  We propose a lightweight and plug-and-play Test-time adaptation framework for correcting Unseen Normal pattErns in graph anomaly detection (GAD)<n>To address semantic confusion, a graph aligner is employed to align the shifted data to the original one at the graph attribute level.<n>Extensive experiments on 10 real-world datasets demonstrate that TUNE significantly enhances the generalizability of pre-trained GAD models to both synthetic and real unseen normal patterns.
  arXiv  Detail & Related papers  (2025-11-10T12:10:05Z)
• Catching Contamination Before Generation: Spectral Kill Switches for Agents [0.0]
  We introduce a diagnostic that uses only the forward pass to emit a binary accept or reject signal during agent execution.<n>The method analyzes token graphs induced by attention and computes two spectral statistics in early layers.<n>We show that a single threshold on the high frequency energy ratio is optimal in the Bayes sense for detecting context inconsistency.
  arXiv  Detail & Related papers  (2025-11-08T02:24:05Z)
• Technical note on Sequential Test-Time Adaptation via Martingale-Driven Fisher Prompting [3.5808917363708743]
  M-FISHER is a method for sequential distribution shift detection and stable adaptation in streaming data.<n>For detection, we construct an exponential martingale from non-conformity scores and apply Ville's inequality to obtain time-uniform guarantees on false alarm control.<n>For adaptation, we show that Fisher-preconditioned updates of prompt parameters implement natural gradient descent on the distributional manifold.
  arXiv  Detail & Related papers  (2025-10-04T15:31:26Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysis [41.09181390655176]
  Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but degrade significantly under textittemporally evolving distribution shifts common in real-world scenarios.<n>We formalize this practical problem as textitContinual-Temporal Test-Time Adaptation (CT-TTA), where test distributions evolve gradually over time.<n>We propose textitBayesTTA, a Bayesian adaptation framework that enforces temporally consistent predictions and dynamically aligns visual representations.
  arXiv  Detail & Related papers  (2025-07-11T14:02:54Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• Near-optimal inference in adaptive linear regression [60.08422051718195]
  Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner. We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
  arXiv  Detail & Related papers  (2021-07-05T21:05:11Z)
• Optimal Change-Point Detection with Training Sequences in the Large and Moderate Deviations Regimes [72.68201611113673]
  This paper investigates a novel offline change-point detection problem from an information-theoretic perspective. We assume that the knowledge of the underlying pre- and post-change distributions are not known and can only be learned from the training sequences which are available.
  arXiv  Detail & Related papers  (2020-03-13T23:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.