When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry

• URL: http://arxiv.org/abs/2604.27656v1
• Date: Thu, 30 Apr 2026 09:50:01 GMT
• Title: When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry
• Abstract summary: We study how network architecture, task similarity, and representational dimensionality jointly shape learning in a sequential task paradigm.<n>Our findings suggest that representational dimensionality acts as a key organizing variable governing when structural separation becomes functionally relevant.
• Score: 4.121514039516763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To preserve previously learned representations, continual learning systems must strike a balance between plasticity, the ability to acquire new knowledge, and stability. This stability-plasticity dilemma affects how representations can be reused across tasks: shared structure enables transfer when tasks are similar but may also induce interference when new learning disrupts existing representations. However, it remains unclear when and why structural separation influences this trade-off. In this study, we examine how network architecture, task similarity, and representational dimensionality jointly shape learning in a sequential task paradigm inspired by transfer-interference studies. We compare a task-partitioned modular recurrent network with a single-module baseline by systematically varying task similarity (low, medium, high) and the scale of weight initialization, which induces different learning regimes that we empirically characterize through the effective dimensionality of the learned representations. We find that architecture has minimal impact in high-dimensional regimes where representations are sufficiently unconstrained to accommodate multiple tasks without strong interference. In contrast, in lower-dimensional (rich) regimes, architectural separation is decisive: modular networks exhibit graded alignment of task-specific subspaces with overlap for similar tasks, partial orthogonalization for moderately dissimilar tasks, and stronger separation for dissimilar tasks. This graded geometry is absent in the single network baseline. Our findings suggest that representational dimensionality acts as a key organizing variable governing when structural separation becomes functionally relevant, and highlight adaptive geometry as a central principle for designing continual learning systems.

Related papers

• Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System [90.64481646805855]
  unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model.<n>We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting.<n>We show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
  arXiv  Detail & Related papers  (2026-09-01T17:59:59Z)
• Hyperbolic Multimodal Continual Learning [90.3699346061366]
  This work provides a theoretical foundation for representation preservation in hyperbolic space.<n>We show that forgetting in hyperbolic continual learning involves both semantic relation drift and hierarchy-related distortion.<n>We propose a principled continual learning framework that preserves essential geometric structure while allowing effective adaptation to new tasks.
  arXiv  Detail & Related papers  (2026-08-10T13:07:26Z)
• Dimensionality Controls When Modularity Helps in Continual Learning [3.841664335605703]
  We study how modular architecture, task similarity, and representational dimensionality jointly shape compositional continual learning.<n>In a high-dimensional "lazy" regime, both architectures achieve similar performance and internal geometry.<n>In a lower-dimensional "rich" regime, modularity becomes decisive.
  arXiv  Detail & Related papers  (2026-06-16T13:10:21Z)
• Uncovering the Latent Potential of Deep Intermediate Representations [16.988545433223933]
  We show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by nave aggregation.<n>We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces.<n>Our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses.
  arXiv  Detail & Related papers  (2026-05-21T20:58:42Z)
• Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance [4.177067537825386]
  This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations.<n>While modern VLMs excel at semantic tasks, they exhibit systematic failures at a more fundamental level.
  arXiv  Detail & Related papers  (2026-04-02T10:02:49Z)
• Cross-Model Semantics in Representation Learning [1.2064681974642195]
  We show that structural regularities induce representational geometry that is more stable under architectural variation.<n>This suggests that certain forms of inductive bias not only support generalization within a model, but also improve the interoperability of learned features across models.
  arXiv  Detail & Related papers  (2025-08-05T16:57:24Z)
• Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning [73.18052192964349]
  We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics.<n>By lifting neural parameters to a measure space and modeling training as Wasserstein gradient flow, we show that under geometric constraints, the parameter measure $mu_t$ undergoes two concurrent phenomena.
  arXiv  Detail & Related papers  (2025-06-26T22:40:30Z)
• The Importance of Being Lazy: Scaling Limits of Continual Learning [60.97756735877614]
  We show that increasing model width is only beneficial when it reduces the amount of feature learning, yielding more laziness.<n>We study the intricate relationship between feature learning, task non-stationarity, and forgetting, finding that high feature learning is only beneficial with highly similar tasks.
  arXiv  Detail & Related papers  (2025-06-20T10:12:38Z)
• Interpretable deformable image registration: A geometric deep learning perspective [9.13809412085203]
  We present a theoretical foundation for designing an interpretable registration framework.<n>We formulate an end-to-end process that refines transformations in a coarse-to-fine fashion.<n>We conclude by showing significant improvement in performance metrics over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-12-17T19:47:10Z)
• Hierarchical Invariance for Robust and Interpretable Vision Tasks at Larger Scales [54.78115855552886]
  We show how to construct over-complete invariants with a Convolutional Neural Networks (CNN)-like hierarchical architecture. With the over-completeness, discriminative features w.r.t. the task can be adaptively formed in a Neural Architecture Search (NAS)-like manner. For robust and interpretable vision tasks at larger scales, hierarchical invariant representation can be considered as an effective alternative to traditional CNN and invariants.
  arXiv  Detail & Related papers  (2024-02-23T16:50:07Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• The Neural Race Reduction: Dynamics of Abstraction in Gated Networks [12.130628846129973]
  We introduce the Gated Deep Linear Network framework that schematizes how pathways of information flow impact learning dynamics. We derive an exact reduction and, for certain cases, exact solutions to the dynamics of learning. Our work gives rise to general hypotheses relating neural architecture to learning and provides a mathematical approach towards understanding the design of more complex architectures.
  arXiv  Detail & Related papers  (2022-07-21T12:01:03Z)
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition [77.95361323613147]
  Current state-of-the-art visual recognition systems rely on pretraining a neural network on a large-scale dataset and finetuning the network weights on a smaller dataset. In this work, we prove that dynamically adapting network architectures tailored for each domain task along with weight finetuning benefits in both efficiency and effectiveness. Our method can be easily generalized to an unsupervised paradigm by replacing supernet training with self-supervised learning in the source domain tasks and performing linear evaluation in the downstream tasks.
  arXiv  Detail & Related papers  (2021-03-31T08:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.