Massive Redundancy in Gradient Transport Enables Sparse Online Learning

• URL: http://arxiv.org/abs/2603.15195v1
• Date: Mon, 16 Mar 2026 12:32:55 GMT
• Title: Massive Redundancy in Gradient Transport Enables Sparse Online Learning
• Abstract summary: Real-time recurrent learning computes exact online gradients by propagating a Jacobian tensor forward through recurrent dynamics.<n>We show that, in the continuous error signal regime, the recurrent Jacobian is massively redundant:propagating through a random 6% of paths.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time recurrent learning (RTRL) computes exact online gradients by propagating a Jacobian tensor forward through recurrent dynamics, but at O(n^4) cost per step. Prior work has sought structured approximations (rank-1 compression, graph-based sparsity, Kronecker factorization). We show that, in the continuous error signal regime, the recurrent Jacobian is massively redundant:propagating through a random 6% of paths (k=4 of n=64) recovers 84 +/- 6% of full RTRL's adaptation ability across five seeds, and the absolute count k=4 remains effective from n=64 to n=256 (6% to 1.6%, recovery 84 to 78%), meaning sparse RTRL becomes relatively cheaper as networks grow. In RNNs, the recovery is selection-invariant (even adversarial path selection works) and exhibits a step-function transition from zero to any nonzero propagation. Spectral analysis reveals the mechanism: the Jacobian is full-rank but near-isotropic (condition numbers 2.6-6.5), so any random subset provides a directionally representative gradient estimate. On chaotic dynamics (Lorenz attractor), sparse propagation is more numerically stable than full RTRL (CV 13% vs. 88%), as subsampling avoids amplifying pathological spectral modes. The redundancy extends to LSTMs (k=4 matches full RTRL) and to transformers via sparse gradient transport (50% head sparsity outperforms the dense reference; 33% is borderline), with higher thresholds reflecting head specialization rather than isotropy. On real primate neural data, sparse RTRL (k=4) adapts online to cross-session electrode drift (80 +/- 11% recovery, 5 seeds), where sparse propagation is again more stable than full RTRL. Without continuous error signal, Jacobian propagation accumulates numerical drift and degrades all RTRL variants, a scope condition for all forward-mode methods. Results hold with SGD (92 +/- 1% recovery), suggesting independence from optimizer choice.

Related papers

• Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale [17.610398782149293]
  In vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal.<n> Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step.<n>It improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery.
  arXiv  Detail & Related papers  (2026-08-18T06:35:26Z)
• DFSC: Error-Controlled Differentiable Mittag-Leffler Propagation for Fractional Scientific Machine Learning [1.6010518212545237]
  PyTorch environment separates known fractional propagation from data-driven corrections.<n>DFSC supports dense, sparse, matrix-free, self-adjoint, generalized, and controlled complex operator paths.
  arXiv  Detail & Related papers  (2026-07-31T05:29:25Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability [0.0]
  CART (Context-Anchored Recurrent Transformer) is a parameter-efficient language model that reuses a single shared core block R times across depth.<n>We evaluate CART on single consumer GPUs in two stages: a 64-configuration screen at 3,000 steps, then 36 configurations (P=6, R in 6,8,10, three seeds) trained for 30,500 steps (1B tokens)<n>Two patterns hold across widths d in 256,512,768,1024: prelude depth P dominates loop count R, and the Stage-1 ranking of R reverses at full training (R=6 becomes best at
  arXiv  Detail & Related papers  (2026-05-31T23:26:27Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Pack only the essentials: Adaptive dictionary learning for kernel ridge regression [69.32538992633842]
  One of the major limits of kernel ridge regression (KRR) is that storing and manipulating the kernel matrix K_n for n samples requires O(n2) space.<n>INK-Estimate is an algorithm that processes the dataset incrementally and updates RLS, effective dimension, and Nystrom approximations on-the-fly.<n>In this paper we introduce SQUEAK, a new algorithm that builds on INK-Estimate but uses unnormalized RLS.
  arXiv  Detail & Related papers  (2026-04-24T09:22:09Z)
• Analysis of Nystrom method with sequential ridge leverage scores [69.32538992633842]
  Large-scale kernel ridge regression (KRR) is limited by the need to store a large kernel matrix K_t.<n>Recent works show that a sampling distribution proportional to the ridge leverage scores (RLSs) provides strong reconstruction guarantees for the approximation.<n>We introduce the INK-ESTIMATE algorithm, that incrementally computes the RLSs estimates.
  arXiv  Detail & Related papers  (2026-04-22T00:49:25Z)
• Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization [0.0]
  We show that gradient descent reliably finds good solutions in non-dimensional network optimization.<n>We identify two regimes separated by a width-dependent dynamical transition.<n>All predictions are validated across 23 experiments.
  arXiv  Detail & Related papers  (2026-04-08T10:41:24Z)
• Inverse Neural Operator for ODE Parameter Optimization [6.148294495010685]
  Inverse Neural Operator (INO) is a framework for recovering hidden ODE parameters from sparse, partial observations.<n>INO outperforms gradient-based and amortized baselines in parameter recovery accuracy while requiring only 0.23s inference time.
  arXiv  Detail & Related papers  (2026-03-12T12:20:23Z)
• BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation" [0.0]
  Activation memory required for exact backpropagation scales linearly with network depth, context length, and feature dimensionality.<n>In this paper, we introduce an efficient backpropagation algorithm that fully decouples activation memory from the batch and sequence dimensions.
  arXiv  Detail & Related papers  (2026-03-05T20:38:25Z)
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts [0.0]
  Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token.<n>Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks.<n>This paper introducesMoE (MLP-Experts), a training-free transformation that restructures the dense in transformer blocks into a static, high-cardinality mixture of experts.
  arXiv  Detail & Related papers  (2025-11-26T06:14:26Z)
• Evolution Strategies at the Hyperscale [57.75314521465674]
  We introduce EGGROLL, an evolution strategies (ES) algorithm designed to scale backprop-free optimization to large population sizes.<n>ES is a set of powerful blackbox optimisation methods that can handle non-differentiable or noisy objectives.<n>EGGROLL overcomes these bottlenecks by generating random matrices $Ain mathbbRmtimes r, Bin mathbbRntimes r$ with $rll min(m,n)$ to form a low-rank matrix perturbation $A Btop$
  arXiv  Detail & Related papers  (2025-11-20T18:56:05Z)
• Exploring Spiking Neural Networks for Binary Classification in Multivariate Time Series at the Edge [0.9282545044546486]
  We present a general framework for training spiking neural networks (SNNs) to perform binary classification on multivariate time series.<n>We apply it to the task of detecting low signal-to-noise ratio radioactive sources in gamma-ray spectral data.<n>The resulting SNNs, with as few as 49 neurons and 66 synapses, achieve a 51.8% true positive rate (TPR) at a false alarm rate of 1/hr.<n> Hardware deployment on the microCaspian neuromorphic platform demonstrates 2mW power consumption and 20.2ms latency.
  arXiv  Detail & Related papers  (2025-10-23T20:52:11Z)
• Environment-Aware Indoor LoRaWAN Path Loss: Parametric Regression Comparisons, Shadow Fading, and Calibrated Fade Margins [3.776919981139063]
  Indoor LoRaWAN propagation is shaped by structural and time-varying context factors.<n>We present an environment-aware, statistically disciplined path loss framework evaluated using leakage-safe cross-validation.
  arXiv  Detail & Related papers  (2025-10-05T20:14:48Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion [55.95767828747407]
  In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model.<n>We present a framework that reduces training variance and provides a provably lower-variance gradient estimator.<n>We also present a practical implementation of this estimator incorporating the loss and sampling procedure through a method we call Orbit Diffusion.
  arXiv  Detail & Related papers  (2025-02-14T03:26:57Z)
• On the Power of Perturbation under Sampling in Solving Extensive-Form Games [56.013335390600524]
  We investigate how perturbation does and does not improve the Follow-the-Regularized-Leader (FTRL) algorithm in solving extensive-form games under sampling.<n>We present a unified framework for textitPerturbed FTRL algorithms and study two variants: PFTRL-KL and PFTRL-RKL.
  arXiv  Detail & Related papers  (2025-01-28T00:29:38Z)
• Convergence Analysis of Real-time Recurrent Learning (RTRL) for a class of Recurrent Neural Networks [0.0]
  Real-time recurrent learning (RTRL) is an online optimization algorithm.<n>It follows the true gradient loss on the data sequence as the number of sequence time steps.<n>One potential application area for RTRL is the analysis of financial data.
  arXiv  Detail & Related papers  (2025-01-14T11:46:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.