Routing without Forgetting

• URL: http://arxiv.org/abs/2603.09576v1
• Date: Tue, 10 Mar 2026 12:23:46 GMT
• Title: Routing without Forgetting
• Abstract summary: We recast continual learning in transformers as a routing problem.<n>Instead of storing or merging task-specific prompts, RwF generates dynamic prompts through single-step associative retrieval.<n>On Split-ImageNet-R and Split-ImageNet-S, RwF outperforms prior prompt-based approaches by a large margin, even in few-shot learning regimes.
• Score: 20.60324059904291
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Continual learning in transformers is commonly addressed through parameter-efficient adaptation: prompts, adapters, or LoRA modules are specialized per task while the backbone remains frozen. Although effective in controlled multi-epoch settings, these approaches rely on gradual gradient-based specialization and struggle in Online Continual Learning (OCL), where data arrive as a non-stationary stream and each sample may be observed only once. We recast continual learning in transformers as a routing problem: under strict online constraints, the model must dynamically select the appropriate representational subspace for each input without explicit task identifiers or repeated optimization. We thus introduce Routing without Forgetting (RwF), a transformer architecture augmented with energy-based associative retrieval layers inspired by Modern Hopfield Networks. Instead of storing or merging task-specific prompts, RwF generates dynamic prompts through single-step associative retrieval over the transformer token embeddings at each layer. Retrieval corresponds to the closed-form minimization of a strictly convex free-energy functional, enabling input-conditioned routing within each forward pass, independently of iterative gradient refinement. Across challenging class-incremental benchmarks, RwF improves over existing prompt-based methods. On Split-ImageNet-R and Split-ImageNet-S, RwF outperforms prior prompt-based approaches by a large margin, even in few-shot learning regimes. These results indicate that embedding energy-based associative routing directly within the transformer backbone provides a principled and effective foundation for OCL.

Related papers

• LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution [73.85089704122315]
  We propose a minimalist framework that performs Video Super-Resolution (VSR) using a frozen Transformer with a lightweight State-Aware Adapter.<n> LiteVSR achieves competitive restoration quality with only 11.25% trainable parameters and 12 GPU-hours of training on a single A100, while maintaining fast sampling (down to a single step) compatibility.
  arXiv  Detail & Related papers  (2026-06-08T09:23:42Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning [83.67663266193308]
  Class-Incremental Learning (CIL) requires a learning system to learn new classes while retaining previously learned knowledge.<n>In real-world scenarios such as autonomous driving, a system trained on urban roads in sunny weather may later need to operate in rural or highway environments.<n>We propose a unified framework to address domain shift and catastrophic forgetting simultaneously.
  arXiv  Detail & Related papers  (2026-05-09T09:49:13Z)
• Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval [61.75093083322011]
  We show that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.<n>Experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.
  arXiv  Detail & Related papers  (2026-05-08T18:55:41Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Hybrid Dual-Path Linear Transformations for Efficient Transformer Architectures [0.0]
  We introduce the Hybrid Dual-Path Linear (HDPL) operator, which decomposes the affine transformation into two topologically distinct pathways.<n> Experiments on the FineWeb-Edu dataset demonstrate that the HDPL architecture outperforms a standard Llama-style baseline.<n>We discuss how the explicit materialization of a probabilistic latent space within the Transformer backbone serves as a vital architectural affordance.
  arXiv  Detail & Related papers  (2026-02-05T20:16:10Z)
• Improving Recursive Transformers with Mixture of LoRAs [2.672804414228544]
  Mixture of LoRAs (MoL) inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN)<n>MoL enables token-conditional weight-space modulation of the shared FFN without untying backbone parameters.<n>ModernALBERT achieves state-of-the-art performance among compact models.
  arXiv  Detail & Related papers  (2025-12-14T23:39:30Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• Proximal Algorithm Unrolling: Flexible and Efficient Reconstruction Networks for Single-Pixel Imaging [45.39911367007956]
  Deep-unrolling and plug-and-play approaches have become the de-facto for single-pixel imaging (SPI) inverse problem.<n>In this paper, we address the challenge of integrating the strengths of both classes of solvers.
  arXiv  Detail & Related papers  (2025-05-29T07:16:57Z)
• Is Attention Required for Transformer Inference? Explore Function-preserving Attention Replacement [13.38679135071682]
  We propose a Function-preserving Attention Replacement framework that replaces all attention blocks in pretrained transformers with learnable sequence-to-sequence modules.<n>We validate FAR on the DeiT vision transformer family and demonstrate that it matches the accuracy of the original models on ImageNet and multiple downstream tasks with reduced parameters and latency.
  arXiv  Detail & Related papers  (2025-05-24T02:23:46Z)
• Towards Transformer-Based Aligned Generation with Self-Coherence Guidance [51.42269790543461]
  We introduce a training-free approach for enhancing alignment in Transformer-based Text-Guided Diffusion Models (TGDMs)<n>Existing TGDMs often struggle to generate semantically aligned images, particularly when dealing with complex text prompts or multi-concept attribute binding challenges.<n>Our method addresses these challenges by directly optimizing cross-attention maps during the generation process.
  arXiv  Detail & Related papers  (2025-03-22T07:03:57Z)
• DLGSANet: Lightweight Dynamic Local and Global Self-Attention Networks for Image Super-Resolution [83.47467223117361]
  We propose an effective lightweight dynamic local and global self-attention network (DLGSANet) to solve image super-resolution. Motivated by the network designs of Transformers, we develop a simple yet effective multi-head dynamic local self-attention (MHDLSA) module to extract local features efficiently. To overcome this problem, we develop a sparse global self-attention (SparseGSA) module to select the most useful similarity values.
  arXiv  Detail & Related papers  (2023-01-05T12:06:47Z)
• Pruning Self-attentions into Convolutional Layers in Single Path [89.55361659622305]
  Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. We propose Single-Path Vision Transformer pruning (SPViT) to efficiently and automatically compress the pre-trained ViTs. Our SPViT can trim 52.0% FLOPs for DeiT-B and get an impressive 0.6% top-1 accuracy gain simultaneously.
  arXiv  Detail & Related papers  (2021-11-23T11:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.