Monte Carlo Functional Regularisation for Continual Learning

• URL: http://arxiv.org/abs/2508.13006v1
• Date: Mon, 18 Aug 2025 15:25:37 GMT
• Title: Monte Carlo Functional Regularisation for Continual Learning
• Authors: Pengcheng Hao, Menghao Waiyan William Zhu, Ercan Engin Kuruoglu,
• Abstract summary: We present a new functional regularisation CL framework, called MCFRCL, which approximates model prediction distributions by Monte Carlo sampling.<n>The proposed MCFRCL is evaluated against multiple benchmark methods on the MNIST and CIFAR datasets.
• Score: 2.2871867623460216
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Continual learning (CL) is crucial for the adaptation of neural network models to new environments. Although outperforming weight-space regularisation approaches, the functional regularisation-based CL methods suffer from high computational costs and large linear approximation errors. In this work, we present a new functional regularisation CL framework, called MCFRCL, which approximates model prediction distributions by Monte Carlo (MC) sampling. Moreover, three continuous distributions are leveraged to capture the statistical characteristics of the MC samples via moment-based methods. Additionally, both the Wasserstein distance and the Kullback-Leibler (KL) distance are employed to construct the regularisation function. The proposed MCFRCL is evaluated against multiple benchmark methods on the MNIST and CIFAR datasets, with simulation results highlighting its effectiveness in both prediction accuracy and training efficiency.

Related papers

• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning [73.93639228235622]
  Continual Learning with foundation models has emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks.<n>Existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks.<n>We propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal.
  arXiv  Detail & Related papers  (2025-01-22T20:00:41Z)
• Scaling Laws for Predicting Downstream Performance in LLMs [75.28559015477137]
  This work focuses on the pre-training loss as a more computation-efficient metric for performance estimation.<n>We present FLP-M, a fundamental approach for performance prediction that addresses the practical need to integrate datasets from multiple sources during pre-training.
  arXiv  Detail & Related papers  (2024-10-11T04:57:48Z)
• Fast training and sampling of Restricted Boltzmann Machines [4.785158987724452]
  We build upon recent theoretical advancements in RBM training, to significantly reduce the computational cost of training. We propose a pre-training phase that encodes the principal components into a low-rank RBM through a convex optimization process. We exploit the continuous and smooth nature of the parameter annealing trajectory to achieve reliable and computationally efficient log-likelihood estimations.
  arXiv  Detail & Related papers  (2024-05-24T09:23:43Z)
• Bayesian Exploration of Pre-trained Models for Low-shot Image Classification [14.211305168954594]
  This work proposes a simple and effective probabilistic model ensemble framework based on Gaussian processes. We achieve the integration of prior knowledge by specifying the mean function with CLIP and the kernel function. We demonstrate that our method consistently outperforms competitive ensemble baselines regarding predictive performance.
  arXiv  Detail & Related papers  (2024-03-30T10:25:28Z)
• Variational Approach for Efficient KL Divergence Estimation in Dirichlet Mixture Models [0.0]
  This study tackles the efficient estimation of Kullback-Leibler (KL) Divergence in Dirichlet Mixture Models (DMM) Previous approaches relied on computationally demanding Monte Carlo methods, motivating our introduction of a novel variational approach.
  arXiv  Detail & Related papers  (2024-03-18T18:14:54Z)
• Online Variational Sequential Monte Carlo [49.97673761305336]
  We build upon the variational sequential Monte Carlo (VSMC) method, which provides computationally efficient and accurate model parameter estimation and Bayesian latent-state inference. Online VSMC is capable of performing efficiently, entirely on-the-fly, both parameter estimation and particle proposal adaptation.
  arXiv  Detail & Related papers  (2023-12-19T21:45:38Z)
• Kalman Filter for Online Classification of Non-Stationary Data [101.26838049872651]
  In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. We introduce a probabilistic Bayesian online learning model by using a neural representation and a state space model over the linear predictor weights. In experiments in multi-class classification we demonstrate the predictive ability of the model and its flexibility to capture non-stationarity.
  arXiv  Detail & Related papers  (2023-06-14T11:41:42Z)
• CLIPood: Generalizing CLIP to Out-of-Distributions [73.86353105017076]
  Contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, but the further adaptation of CLIP on downstream tasks undesirably degrades OOD performances. We propose CLIPood, a fine-tuning method that can adapt CLIP models to OOD situations where both domain shifts and open classes may occur on unseen test data. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.
  arXiv  Detail & Related papers  (2023-02-02T04:27:54Z)
• Low-variance estimation in the Plackett-Luce model via quasi-Monte Carlo sampling [58.14878401145309]
  We develop a novel approach to producing more sample-efficient estimators of expectations in the PL model. We illustrate our findings both theoretically and empirically using real-world recommendation data from Amazon Music and the Yahoo learning-to-rank challenge.
  arXiv  Detail & Related papers  (2022-05-12T11:15:47Z)
• On Continual Model Refinement in Out-of-Distribution Data Streams [64.62569873799096]
  Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams. Existing continual learning (CL) problem setups cannot cover such a realistic and complex scenario. We propose a new CL problem formulation dubbed continual model refinement (CMR)
  arXiv  Detail & Related papers  (2022-05-04T11:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.