MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data

• URL: http://arxiv.org/abs/2605.14364v3
• Date: Wed, 20 May 2026 00:19:26 GMT
• Title: MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data
• Abstract summary: Continual learning requires models to adapt to new data while preserving previously acquired knowledge.<n>Most existing approaches address this challenge by modifying model parameters or architectures in a supervised, task-specific manner.<n>We propose MoRe, a framework that identifies modularity in the representation itself rather than allocating it at the architectural level.
• Score: 17.462860448981953
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning requires models to adapt to new data while preserving previously acquired knowledge. At its core, this challenge can be viewed as principled one-step adaptation: incorporating new information with minimal interference to existing representations. Most existing approaches address this challenge by modifying model parameters or architectures in a supervised, task-specific manner. However, the underlying issue is representational: tasks require distinct yet structured representations that can be selectively updated without disrupting representations, while structure should reflect intrinsic organization in the data rather than task boundaries. In sequential data, time-delayed dependencies provide a natural signal for uncovering this organization, revealing how fundamental representations give rise to more specific ones. Inspired by the modular organization of the human brain, we propose MoRe, a framework that identifies modularity in the representation itself rather than allocating it at the architectural level. MoRe decomposes knowledge into a hierarchy of fundamental and specific modules with identifiability guarantees, enabling principled module reuse, alignment, and expansion during adaptation while preserving old modules by construction. Experiments on synthetic benchmarks and real-world LLM activations demonstrate interpretable hierarchical structure, improved plasticity-stability trade-offs, suggesting MoRe as a principled foundation for continual adaptation

Related papers

• What Is Worth Representing? Representational Empowerment for Continual Model Construction [2.184419106993377]
  We propose Representational Empowerment (RepEmp) to score candidate elements by how much they expand the agent's future capacity to model and plan.<n>We realize the framework as a hierarchical Curator-Actor architecture and test it across three experiments.
  arXiv  Detail & Related papers  (2026-09-02T09:05:42Z)
• Improving the Understandability of Conceptual Models via Abstract Notation Engineering [1.513674779059522]
  We propose abstract notation engineering as a language-agnostic method for replacing low-level, semantically transparent constructs.<n>We instantiate it for Dynamic Condition Response (DCR) graphs, where common workflow patterns require elaborate low-level configurations.<n>The results of our empirical validation show that DeCleaR improves perceived empirical quality, pragmatic quality, and user preference over standard DCR graphs.
  arXiv  Detail & Related papers  (2026-07-31T15:43:34Z)
• Pre-Forgettable Models: Prompt Learning as a Native Mechanism for Unlearning [9.512928441517811]
  Foundation models have transformed multimedia analysis by enabling robust and transferable representations across diverse modalities and tasks.<n>Traditional unlearning approaches, including retraining, activation editing, or distillation, are often expensive, fragile, and ill-suited for real-time or continuously evolving systems.<n>We introduce a prompt-based learning framework that unifies knowledge acquisition and removal within a single training phase.
  arXiv  Detail & Related papers  (2025-09-05T13:28:04Z)
• Integrated Structural Prompt Learning for Vision-Language Models [15.002501540565781]
  In this paper, we propose an Integrated Structural Prompt (ISP) for Vision-Language Models (VLMs)<n>ISP introduces self-structural and cross-structural prompt modules to model the structural relationships between learnable prompts and frozen tokens.<n>ISP achieves competitive performance against state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-07-08T04:59:58Z)
• Intervening to learn and compose disentangled representations [8.452837716541705]
  We propose a new approach to training arbitrarily expressive generative models that simultaneously learn disentangled latent structure.<n>This is accomplished by adding a simple decoder-only module to the head of an existing decoder block that can be arbitrarily complex.<n>Inspired by the notion of intervention in causal graphical models, our module selectively modifies its architecture during training, allowing it to learn a compact joint model over different contexts.
  arXiv  Detail & Related papers  (2025-07-07T08:30:27Z)
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction [57.19302613163439]
  We introduce neural network reprogrammability as a unifying framework for model adaptation.<n>We present a taxonomy that categorizes such information manipulation approaches across four key dimensions.<n>We also analyze remaining technical challenges and ethical considerations.
  arXiv  Detail & Related papers  (2025-06-05T05:42:27Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)
• Modular Deep Learning [120.36599591042908]
  Transfer learning has recently become the dominant paradigm of machine learning. It remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference. Modular deep learning has emerged as a promising solution to these challenges.
  arXiv  Detail & Related papers  (2023-02-22T18:11:25Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)
• Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models [61.480085460269514]
  We propose a framework for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models. We use this framework to build ModularQA, a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator.
  arXiv  Detail & Related papers  (2020-09-01T23:45:42Z)
• S2RMs: Spatially Structured Recurrent Modules [105.0377129434636]
  We take a step towards exploiting dynamic structure that are capable of simultaneously exploiting both modular andtemporal structures. We find our models to be robust to the number of available views and better capable of generalization to novel tasks without additional training.
  arXiv  Detail & Related papers  (2020-07-13T17:44:30Z)
• Structure by Architecture: Structured Representations without Regularization [31.75200752252397]
  We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling. We design a novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization. We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including generation, disentanglement, and extrapolation.
  arXiv  Detail & Related papers  (2020-06-14T04:37:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.