Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization

• URL: http://arxiv.org/abs/2607.13432v1
• Date: Wed, 15 Jul 2026 04:31:20 GMT
• Title: Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization
• Abstract summary: Plasticity is a neural network's ability to adapt to new tasks.<n>Existing measures, such as effective rank, dead neuron fraction, and weight norm, lack theoretical grounding.<n>We introduce local redundancy, an information-theoretic measure derived from universal compression theory.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Plasticity -- a neural network's ability to adapt to new tasks -- is critical for continual and transfer learning. Existing measures, such as effective rank, dead neuron fraction, and weight norm, lack theoretical grounding and correlate poorly with performance on new tasks. We introduce local redundancy, an information-theoretic measure derived from universal compression theory. We define local redundancy as the worst-case redundancy of a local model family -- parameters in an infinitesimal neighborhood along gradient directions -- and show this is a principled measure of plasticity. Although local redundancy is intractable to compute exactly, we prove that the expected squared gradient norm on a synthetic memorization task provides an efficiently computable lower bound. Experiments on continual image classification and time series transfer learning demonstrate that local redundancy predicts downstream performance better than existing measures and enables pretraining checkpoint selection where validation loss plateaus.

Related papers

• Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networks [0.0]
  Continual learning research attempts to conserve two fundamental capabilities: new knowledge acquisition and the preservation of previously acquired knowledge.<n>Recent research has identified multiple mechanisms underlying this phenomenon, including neuron saturation, norm growth, and loss of useful curvature directions.<n> Adaptive reset-based interventions have emerged as practical solutions to restore trainability.
  arXiv  Detail & Related papers  (2026-05-07T18:42:22Z)
• Rethinking Plasticity in Deep Reinforcement Learning [3.18807491942654]
  This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL)<n>We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks.<n>Our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex domains.
  arXiv  Detail & Related papers  (2026-03-22T11:27:16Z)
• Evaluating the Efficiency of Latent Spaces via the Coupling-Matrix [0.5013248430919224]
  We introduce a redundancy index, denoted rho(C), that directly quantifies inter-dimensional dependencies.<n>Low rho(C) reliably predicts high classification accuracy or low reconstruction error, while elevated redundancy is associated with performance collapse.<n>We show that Tree-structured Parzen Estimators (TPE) preferentially explore low-rho regions, suggesting that rho(C) can guide neural architecture search and serve as a redundancy-aware regularization target.
  arXiv  Detail & Related papers  (2025-09-08T03:36:47Z)
• Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning [56.87989363424]
  We show that a low-rank structure naturally emerges in the shifted successor measure.<n>We quantify the amount of shift needed for effective low-rank approximation and estimation.
  arXiv  Detail & Related papers  (2025-09-05T15:48:20Z)
• Doubly Robust Causal Effect Estimation under Networked Interference via Targeted Learning [24.63284452991301]
  We propose a doubly robust causal effect estimator under networked interference. Specifically, we generalize the targeted learning technique into the networked interference setting. We devise an end-to-end causal effect estimator by transforming the identified theoretical condition into a targeted loss.
  arXiv  Detail & Related papers  (2024-05-06T10:49:51Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [86.99017195607077]
  We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents.<n>Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
  arXiv  Detail & Related papers  (2024-04-04T06:24:11Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Convolutional generative adversarial imputation networks for spatio-temporal missing data in storm surge simulations [86.5302150777089]
  Generative Adversarial Imputation Nets (GANs) and GAN-based techniques have attracted attention as unsupervised machine learning methods. We name our proposed method as Con Conval Generative Adversarial Imputation Nets (Conv-GAIN)
  arXiv  Detail & Related papers  (2021-11-03T03:50:48Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• Unsupervised Metric Relocalization Using Transform Consistency Loss [66.19479868638925]
  Training networks to perform metric relocalization traditionally requires accurate image correspondences. We propose a self-supervised solution, which exploits a key insight: localizing a query image within a map should yield the same absolute pose, regardless of the reference image used for registration. We evaluate our framework on synthetic and real-world data, showing our approach outperforms other supervised methods when a limited amount of ground-truth information is available.
  arXiv  Detail & Related papers  (2020-11-01T19:24:27Z)
• Two Routes to Scalable Credit Assignment without Weight Symmetry [25.485770386464186]
  We investigate a recently proposed local learning rule that yields competitive performance with backpropagation. We find a performance and stability gap between this local rule and backpropagation that widens with increasing model depth. We then investigate several non-local learning rules that relax the need for instantaneous weight transport into a more biologically-plausible "weight estimation" process.
  arXiv  Detail & Related papers  (2020-02-28T18:39:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.