Dimensionality Controls When Modularity Helps in Continual Learning

• URL: http://arxiv.org/abs/2606.17889v1
• Date: Tue, 16 Jun 2026 13:10:21 GMT
• Title: Dimensionality Controls When Modularity Helps in Continual Learning
• Abstract summary: We study how modular architecture, task similarity, and representational dimensionality jointly shape compositional continual learning.<n>In a high-dimensional "lazy" regime, both architectures achieve similar performance and internal geometry.<n>In a lower-dimensional "rich" regime, modularity becomes decisive.
• Score: 3.841664335605703
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compositional learning systems must balance plasticity, the ability to acquire new knowledge, with stability, the preservation of previously learned components, especially when tasks share structure and risk interference. We study how modular architecture, task similarity, and representational dimensionality jointly shape compositional continual learning in a sequential A-B-A paradigm, comparing a task-partitioned recurrent network to a single-network baseline while inducing high- and low-dimensional regimes via weight-scale manipulations. In a high-dimensional "lazy" regime, both architectures achieve similar performance and internal geometry, suggesting that explicit modular structure has little impact when representations are weakly constrained. In a lower-dimensional "rich" regime, modularity becomes decisive: the modular network develops graded task-specific subspaces that overlap for similar tasks, partially align for moderately dissimilar tasks, and separate for dissimilar tasks, yielding a more compositional and interpretable organization than the single network. These findings identify the representational regime induced by initialization scale, which co-varies with representational dimensionality, as a key factor governing when compositional, modular structure is functionally beneficial in continual learning, and support viewing safety and robustness as problems of adaptive allocation of representational subspaces rather than fixed separation versus sharing.

Related papers

• Modular Diffusion Models for Structured Visual Recognition [52.38687800325397]
  We present Modular Diffusion Models (MDMs), a simple and novel framework that learns a distribution over structured outputs for a given input image.<n>MDMs decompose the diffusion process into distinct, task-specific modules, each focused on capturing a different aspect of the structured information space.<n>This modular design allows each component to be learned independently, with seamless integration at inference without additional training.
  arXiv  Detail & Related papers  (2026-06-21T22:38:35Z)
• MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data [17.462860448981953]
  Continual learning requires models to adapt to new data while preserving previously acquired knowledge.<n>Most existing approaches address this challenge by modifying model parameters or architectures in a supervised, task-specific manner.<n>We propose MoRe, a framework that identifies modularity in the representation itself rather than allocating it at the architectural level.
  arXiv  Detail & Related papers  (2026-05-14T04:46:54Z)
• When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry [4.121514039516763]
  We study how network architecture, task similarity, and representational dimensionality jointly shape learning in a sequential task paradigm.<n>Our findings suggest that representational dimensionality acts as a key organizing variable governing when structural separation becomes functionally relevant.
  arXiv  Detail & Related papers  (2026-04-30T09:50:01Z)
• Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI [43.38839324900098]
  We introduce a framework in which both the spatial placement and internal segmentation of foundation models are elevated to runtime-resolved constructs.<n>We introduce architectural and algorithmic components, along with a representative use case in 6G multi-access edge computing.
  arXiv  Detail & Related papers  (2025-11-30T19:16:30Z)
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning [58.533203990515034]
  Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL)<n>We show that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements.<n>We finally distill these insights into Module-Specific Training (MST), a practical framework that exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.
  arXiv  Detail & Related papers  (2025-10-14T03:03:08Z)
• Foundation Model for Skeleton-Based Human Action Understanding [56.89025287217221]
  This paper presents a Unified Skeleton-based Dense Representation Learning framework.<n>USDRL consists of a Transformer-based Dense Spatio-Temporal (DSTE), Multi-Grained Feature Decorrelation (MG-FD), and Multi-Perspective Consistency Training (MPCT)
  arXiv  Detail & Related papers  (2025-08-18T02:42:16Z)
• Cross-Model Semantics in Representation Learning [1.2064681974642195]
  We show that structural regularities induce representational geometry that is more stable under architectural variation.<n>This suggests that certain forms of inductive bias not only support generalization within a model, but also improve the interoperability of learned features across models.
  arXiv  Detail & Related papers  (2025-08-05T16:57:24Z)
• Interpretable deformable image registration: A geometric deep learning perspective [9.13809412085203]
  We present a theoretical foundation for designing an interpretable registration framework.<n>We formulate an end-to-end process that refines transformations in a coarse-to-fine fashion.<n>We conclude by showing significant improvement in performance metrics over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-12-17T19:47:10Z)
• Independent Modular Networks [3.10678167047537]
  Monolithic neural networks dismiss the compositional nature of data generation processes. We propose a modular network architecture that splits the modules into roles. We also provide regularizations that improve the resiliency of the modular network to the problem of module collapse.
  arXiv  Detail & Related papers  (2023-06-02T07:29:36Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition [140.18376685167857]
  A simple yet effective multi-scale semantics-guided neural network is proposed for skeleton-based action recognition. MS-SGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets.
  arXiv  Detail & Related papers  (2021-11-07T03:50:50Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• S2RMs: Spatially Structured Recurrent Modules [105.0377129434636]
  We take a step towards exploiting dynamic structure that are capable of simultaneously exploiting both modular andtemporal structures. We find our models to be robust to the number of available views and better capable of generalization to novel tasks without additional training.
  arXiv  Detail & Related papers  (2020-07-13T17:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.