The RG-Flow Transformer: Encoding Scale-Free Dynamics in Scarce EEG

• URL: http://arxiv.org/abs/2607.11950v2
• Date: Thu, 16 Jul 2026 17:07:32 GMT
• Title: The RG-Flow Transformer: Encoding Scale-Free Dynamics in Scarce EEG
• Abstract summary: We ask whether a transformer endowed with an explicit renormalization-group inductive bias the RG-Flow Transformer has an advantage over a parameter-matched vanilla transformer on emphreal, scarce EEG.<n>We benchmark RG-Flow against a param-matched vanilla transformer and a hierarchy-only ablation on 5-class AASM sleep staging.<n>Across $5$ subjects and $5$ seeds under leave-one-subject-out cross-validation, RG-Flow and the vanilla transformer are statistically indistinguishable on 5-class staging.
• Score: 0.09229852843814058
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Brain field potentials are scale-free: their power spectra follow a $1/f^β$ law whose aperiodic exponent $β$ tracks cortical state, and sleep depth in particular is a shift in $β$. We ask whether a transformer endowed with an explicit renormalization-group (RG) inductive bias the RG-Flow Transformer, which couples ordinary self-attention to a scale-aware stream with a learnable anomalous dimension $γ$, block-spin coarse-graining, and an entropy-gated synchronization bridge has an advantage over a parameter-matched vanilla transformer on \emph{real, scarce} EEG. Using the PhysioNet Sleep-EDF corpus with a strict leakage-free by-subject hold-out, we (i) benchmark RG-Flow against a param-matched vanilla transformer and a hierarchy-only ablation on 5-class AASM sleep staging, (ii) sweep the per-subject data budget to look for the inductive-bias crossover predicted when data are scarce, and (iii) test whether RG-Flow's learned $γ$ tracks the measured spectral exponent $β$ out-of-sample a quantity the vanilla model does not possess. Across $5$ subjects and $5$ seeds under leave-one-subject-out cross-validation, RG-Flow and the vanilla transformer are statistically indistinguishable on 5-class staging (77.3\% vs 77.0\% accuracy; paired $p=0.294$), and the predicted scarce-data crossover does not appear: vanilla is numerically ahead at every data-limited budget. What does separate the models is interpretability RG-Flow recovers the continuous spectral exponent out-of-sample ($β$-recovery $R^2 = 0.416$), a capability the vanilla architecture has no analogue for.

Related papers

• The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning [3.860898721527665]
  LoRA fine-tuning can create intruder dimensions that drive catastrophic forgetting.<n>No theory has predicted, layer by layer on measured spectra, when they appear.<n>Full fine-tuning disperses its update far below the threshold of every layer.<n>Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting.
  arXiv  Detail & Related papers  (2026-07-26T15:17:41Z)
• Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't [78.36679389968772]
  We find that padded $textL$ constantprecision transformers are surprisingly robust to expressivity.<n>$logd-looped constant-precision transformers reach $textFO-uniform TCd$, and growing-precision ones reach $textFO-uniform TCd$.
  arXiv  Detail & Related papers  (2026-05-28T19:58:43Z)
• Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension [13.891522069967507]
  Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57times$ energy efficiency on neuromorphic hardware.<n>This paper establishes the first comprehensive expressivity theory for spiking self-attention.
  arXiv  Detail & Related papers  (2026-04-17T07:15:53Z)
• Dimensional Criticality at Grokking Across MLPs and Transformers [2.652953665748039]
  Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems.<n>We introduce textbfTDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe.<n>Across Transformers trained on modular addition and generalizations trained on XOR, we discover a localized crossing of the diffusion baseline $D=1$.
  arXiv  Detail & Related papers  (2026-04-06T13:43:20Z)
• Gated Removal of Normalization in Transformers Enables Stable Training and Efficient Inference [2.1665689529884697]
  We introduce TaperNorm, a drop-in replacement for RMSNorm/LayerNorm that behaves exactly like the standard normalizer early in training.<n>A single global gate is held at $g=1$ during gate warmup, used to calibrate the scaling branch via EMAs, and then cosine-decayed to $g=0$.<n>On an efficiency microbenchmark, folding internal scalings yields up to $1.22times$ higher throughput in last-token logits mode.
  arXiv  Detail & Related papers  (2026-02-11T01:40:34Z)
• Reevaluating Convolutional Neural Networks for Spectral Analysis: A Focus on Raman Spectroscopy [1.1545092788508224]
  We evaluate one-dimensional convolutional neural networks (CNNs) and report four advances.<n> compact CNNs surpass $k$-nearest-neighbors on handcrafted features.<n>label-efficient learning: semi-supervised generative adversarial networks and contrastive pretraining raise accuracy by up to $11%$ with only $10%$ labels.<n>This workflow involves training on raw spectra, tuning pooling, adding semi-supervision when labels are scarce, and fine-tuning lightly for new targets.
  arXiv  Detail & Related papers  (2025-09-30T09:01:38Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• On the Role of Depth and Looping for In-Context Learning with Task Diversity [69.4145579827826]
  We study in-context learning for linear regression with diverse tasks. We show that multilayer Transformers are not robust to even distributional shifts as small as $O(e-L)$ in Wasserstein distance.
  arXiv  Detail & Related papers  (2024-10-29T03:27:56Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification [9.312432667108691]
  Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial Transformer. In this paper, we innovatively replace vanilla SSA with spike-form Fourier Transform, Wavelet Transform, and their combinations, based on a key hypothesis that both of them use a set of basis functions for information transformation. The FWformer can achieve comparable or even higher accuracies ($0.4%$-$1.5%$), higher running speed ($9%$-$51%$ for training and $19%$-$70%$ for inference
  arXiv  Detail & Related papers  (2024-03-27T03:31:16Z)
• p-Laplacian Transformer [7.2541371193810384]
  $p$-Laplacian regularization, rooted in graph and image signal processing, introduces a parameter $p$ to control the regularization effect on these data. We first show that the self-attention mechanism obtains the minimal Laplacian regularization. We then propose a novel class of transformers, namely the $p$-Laplacian Transformer (p-LaT)
  arXiv  Detail & Related papers  (2023-11-06T16:25:56Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers [71.31712741938837]
  We show that sparse Transformers with only $O(n)$ connections per attention layer can approximate the same function class as the dense model with $n2$ connections. We also present experiments comparing different patterns/levels of sparsity on standard NLP tasks.
  arXiv  Detail & Related papers  (2020-06-08T18:30:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.