Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning

• URL: http://arxiv.org/abs/2510.16877v1
• Date: Sun, 19 Oct 2025 15:21:50 GMT
• Title: Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning
• Abstract summary: Fly-CL is a bio-inspired framework compatible with a wide range of pretrained backbones.<n>We show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity.
• Score: 42.58175645355976
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as a similarity-matching problem to mitigate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearity in the similarity-matching stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with a wide range of pretrained backbones. Fly-CL substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods. We theoretically show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL's effectiveness in addressing this challenge through a biologically inspired design. Code is available at https://github.com/gfyddha/Fly-CL.

Related papers

• Towards Realistic Class-Incremental Learning with Free-Flow Increments [17.558920457942936]
  We propose a model-agnostic framework for robust CIL learning under free-flow arrivals.<n>It comprises a class-wise mean (CWM) objective that replaces sample frequency weighted loss with uniformly aggregated class-conditional supervision.<n>Experiments confirm a clear performance degradation across various CIL baselines under FFCIL.
  arXiv  Detail & Related papers  (2026-04-03T06:19:58Z)
• A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation [31.097911935522674]
  Most fine-grained fashion image retrieval methods assume a static setting, requiring full retraining when new attributes appear.<n>We propose a multihead continual learning framework for fine-grained fashion image retrieval with contrastive learning and exponential moving average distillation (MCL-FIR)<n>MCL-FIR adopts a multi-head design to accommodate evolving classes across increments, reformulates triplet inputs into doublets with InfoNCE for simpler and more effective training, and employs EMA distillation for efficient knowledge transfer.
  arXiv  Detail & Related papers  (2026-03-21T04:58:02Z)
• SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators [0.4369550829556577]
  New experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data.<n>This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting.<n>We introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL.
  arXiv  Detail & Related papers  (2026-03-20T18:30:38Z)
• SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism [2.4349098308669594]
  This paper introduces a new training methodology, Supervised Contrastive Parallel Learning ( SCPL), that addresses this issue by decoupling BP and transforming a long gradient flow into multiple short ones.<n> Detailed experiments are presented to demonstrate the efficiency and effectiveness of our model compared to BP, Early Exit, GPipe, and Associated Learning (AL), a state-of-the-art method for decoupling backpropagation.<n> SCPL provides a practical pathway for organizations to develop and deploy advanced information systems more cost-effectively and with greater agility.
  arXiv  Detail & Related papers  (2026-01-20T09:19:30Z)
• FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning [63.20028888397869]
  FOREVER (FORgEtting curVe-inspired mEmory) is a novel framework that aligns replay schedules with a model-centric notion of time.<n>Building on this approach, FOREVER incorporates a forgetting curve-based replay scheduler to determine when to replay and an intensity-aware regularization mechanism to adaptively control how to replay.
  arXiv  Detail & Related papers  (2026-01-07T13:55:14Z)
• DiRL: An Efficient Post-Training Framework for Diffusion Language Models [54.405206032785706]
  Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models.<n>Existing methods suffer from computational inefficiency and objective mismatches between training and inference.<n>We introduce DiRL, an efficient post-training framework that tightly integrates FlexAttention-accelerated blockwise training with LMDeploy-optimized inference.
  arXiv  Detail & Related papers  (2025-12-23T08:33:19Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• Causal Disentanglement and Cross-Modal Alignment for Enhanced Few-Shot Learning [11.752632557524969]
  Causal CLIP Adapter (CCA) is a novel framework that explicitly disentangles visual features extracted from CLIP.<n>Our method consistently outperforms state-of-the-art approaches in terms of few-shot performance and robustness to distributional shifts.
  arXiv  Detail & Related papers  (2025-08-05T05:30:42Z)
• AmorLIP: Efficient Language-Image Pretraining via Amortization [52.533088120633785]
  Contrastive Language-Image Pretraining (CLIP) has demonstrated strong zero-shot performance across diverse downstream text-image tasks.<n>We propose AmorLIP, an efficient CLIP pretraining framework that amortizes expensive computations involved in contrastive learning through lightweight neural networks.
  arXiv  Detail & Related papers  (2025-05-25T05:30:37Z)
• CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learning [6.738409533239947]
  Deep neural networks struggle with catastrophic forgetting when learning tasks sequentially.<n>Recent approaches constrain updates to subspaces using gradient projection.<n>We propose Conceptor-based gradient projection for Deep Continual Learning (CODE-CL)
  arXiv  Detail & Related papers  (2024-11-21T22:31:06Z)
• TS-ACL: Closed-Form Solution for Time Series-oriented Continual Learning [16.270548433574465]
  Time series class-incremental learning faces two major challenges: catastrophic forgetting and intra-class variations.<n>We propose TS-ACL, which leverages a gradient-free closed-form solution to avoid the catastrophic forgetting problem.<n>It also provides privacy protection and efficiency.
  arXiv  Detail & Related papers  (2024-10-21T12:34:02Z)
• LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning [103.45785408116146]
  Continual learning (CL) aims to train a model that can solve multiple tasks presented sequentially.<n>Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks.<n>However, such methods lack theoretical guarantees, making them prone to unexpected failures.<n>We aim to bridge this gap by designing a simple CL method that is theoretically sound and highly performant.
  arXiv  Detail & Related papers  (2024-10-01T12:58:37Z)
• Adaptive Multi-Scale Decomposition Framework for Time Series Forecasting [26.141054975797868]
  We propose a novel Adaptive Multi-Scale Decomposition (AMD) framework for time series forecasting.<n>Our framework decomposes time series into distinct temporal patterns at multiple scales, leveraging the Multi-Scale Decomposable Mixing (MDM) block.<n>Our approach effectively models both temporal and channel dependencies and utilizes autocorrelation to refine multi-scale data integration.
  arXiv  Detail & Related papers  (2024-06-06T05:27:33Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.