Sequential Learning and Catastrophic Forgetting in Differentiable Resistor Networks

• URL: http://arxiv.org/abs/2605.01383v1
• Date: Sat, 02 May 2026 11:15:57 GMT
• Title: Sequential Learning and Catastrophic Forgetting in Differentiable Resistor Networks
• Abstract summary: We investigate sequential learning in differentiable resistor networks governed by Kirchhoff's laws.<n>We show that forgetting is controlled by task conflict and by the degree of adaptation to the new task.<n>These results position differentiable resistor networks as compact, physically interpretable testbeds for studying continual learning in tunable matter.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable physical networks provide a simple setting in which learning can be studied through the interaction between trainable parameters and physical equilibrium constraints. We investigate sequential learning in differentiable resistor networks governed by Kirchhoff's laws. Although individual input--output mappings can be learned by gradient-based adjustment of edge conductances, sequential training on conflicting tasks produces catastrophic forgetting. We show that forgetting is controlled by task conflict and by the degree of adaptation to the new task. Uniform anchoring and normalised gradient-weighted anchoring reduce forgetting only by increasing the final loss on the new task, giving a clear forgetting--adaptation trade-off. We also show that forgetting is associated with localised conductance changes on high-current edges, giving a physical interpretation as reconfiguration of dominant transport pathways. Broader random-task ensembles show that the strongest forgetting occurs when the second task reverses the output ordering imposed by the first task. Finally, comparisons across Erdős--Rényi, small-world, scale-free, and random-geometric graph ensembles show that topology changes the forgetting--adaptation balance. These results position differentiable resistor networks as compact, physically interpretable testbeds for studying continual learning in tunable matter.

Related papers

• Interference and Retention in Continual Learning [0.0]
  We show that forgetting from learning a new task is exactly the interference energy induced on the old task.<n>IGFA is a replay-free, Fisher-free method that shares directions when tasks align and protects them when they conflict.
  arXiv  Detail & Related papers  (2026-07-10T08:48:23Z)
• Task Switching Without Forgetting via Proximal Decoupling [43.88767637201092]
  In continual learning, the primary challenge is to learn new information without forgetting old knowledge.<n>A common solution addresses this trade-off through regularization, penalizing changes to parameters critical for previous tasks.<n>We propose a different approach that separates task learning from stability enforcement via operator splitting.
  arXiv  Detail & Related papers  (2026-04-20T21:28:13Z)
• On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking [49.1352577985191]
  We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task.<n>Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics.
  arXiv  Detail & Related papers  (2026-02-18T20:25:13Z)
• The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions [51.68215326304272]
  We show that even small perturbations reliably cause otherwise identical training trajectories to diverge-an effect that diminishes rapidly over training time.<n>Our findings provide insights into neural network training stability, with practical implications for fine-tuning, model merging, and diversity of model ensembles.
  arXiv  Detail & Related papers  (2025-06-16T08:35:16Z)
• How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias [48.9399496805422]
  We focus on two representative tasks in the category of regular language recognition, known as even pairs' and parity check'<n>Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks.
  arXiv  Detail & Related papers  (2025-05-02T00:07:35Z)
• Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts [1.9458156037869137]
  convergent learning is the degree to which neural systems arrive at similar internal representations.<n>We present a large-scale audit of convergent learning spanning dozens of vision models and thousands of layer-pair comparisons.<n>Findings fill critical gaps in our understanding of representational convergence, with implications for neuroscience and AI.
  arXiv  Detail & Related papers  (2025-02-26T00:04:24Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Slimmable Networks for Contrastive Self-supervised Learning [69.9454691873866]
  Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. We introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers. A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks.
  arXiv  Detail & Related papers  (2022-09-30T15:15:05Z)
• Gradient-trained Weights in Wide Neural Networks Align Layerwise to Error-scaled Input Correlations [11.176824373696324]
  We derive the layerwise weight dynamics of infinite-width neural networks with nonlinear activations trained by gradient descent. We formulate backpropagation-free learning rules, named Align-zero and Align-ada, that theoretically achieve the same alignment as backpropagation.
  arXiv  Detail & Related papers  (2021-06-15T21:56:38Z)
• Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision [57.14468881854616]
  We propose an auxiliary training objective that improves the generalization capabilities of neural networks. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
  arXiv  Detail & Related papers  (2020-04-20T02:47:49Z)
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss [0.0]
  Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. We analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations.
  arXiv  Detail & Related papers  (2020-02-11T15:42:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.