Learning New Concepts, Remembering the Old: Continual Learning for Multimodal Concept Bottleneck Models

• URL: http://arxiv.org/abs/2411.17471v2
• Date: Mon, 04 Aug 2025 21:11:44 GMT
• Title: Learning New Concepts, Remembering the Old: Continual Learning for Multimodal Concept Bottleneck Models
• Authors: Songning Lai, Mingqian Liao, Zhangyi Hu, Jiayu Yang, Wenshuo Chen, Hongru Xiao, Jianheng Tang, Haicheng Liao, Yutao Yue,
• Abstract summary: Concept Bottleneck Models (CBMs) enhance interpretability of AI systems by bridging visual input with human-understandable concepts.<n>Existing CBMs typically assume static datasets, which limits their adaptability to real-world, continuously evolving multimodal data streams.<n>We define a novel continual learning task for CBMs: simultaneously handling concept-incremental and class-incremental learning.
• Score: 11.716449233474451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of AI systems, particularly by bridging visual input with human-understandable concepts, effectively acting as a form of multimodal interpretability model. However, existing CBMs typically assume static datasets, which fundamentally limits their adaptability to real-world, continuously evolving multimodal data streams. To address this, we define a novel continual learning task for CBMs: simultaneously handling concept-incremental and class-incremental learning. This task requires models to continuously acquire new concepts (often representing cross-modal attributes) and classes while robustly preserving previously learned knowledge. To tackle this challenging problem, we propose CONceptual Continual Incremental Learning (CONCIL), a novel framework that fundamentally re-imagines concept and decision layer updates as linear regression problems. This reformulation eliminates the need for gradient-based optimization, thereby effectively preventing catastrophic forgetting. Crucially, CONCIL relies solely on recursive matrix operations, rendering it highly computationally efficient and well-suited for real-time and large-scale multimodal data applications. Experimental results compellingly demonstrate that CONCIL achieves "absolute knowledge memory" and significantly surpasses the performance of traditional CBM methods in both concept- and class-incremental settings, thus establishing a new paradigm for continual learning in CBMs, particularly valuable for dynamic multimodal understanding.

Related papers

• Bisecle: Binding and Separation in Continual Learning for Video Language Understanding [11.710573955384511]
  We propose Bisecle for video-language continual learning, inspired by the rapid Binding and pattern separation mechanisms in the hippocampus.<n>Bisecle mitigates forgetting and enhances cross-task generalization on several VideoQA benchmarks.
  arXiv  Detail & Related papers  (2025-07-01T06:28:57Z)
• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• Towards Better Generalization and Interpretability in Unsupervised Concept-Based Models [9.340843984411137]
  This paper introduces a novel unsupervised concept-based model for image classification, named Learnable Concept-Based Model (LCBM)<n>We demonstrate that LCBM surpasses existing unsupervised concept-based models in generalization capability and nearly matches the performance of black-box models.<n>Despite the use of concept embeddings, we maintain model interpretability by means of a local linear combination of concepts.
  arXiv  Detail & Related papers  (2025-06-02T16:26:41Z)
• Self-Controlled Dynamic Expansion Model for Continual Learning [10.447232167638816]
  This paper introduces an innovative Self-Controlled Dynamic Expansion Model (SCDEM)<n>SCDEM orchestrates multiple trainable pre-trained ViT backbones to furnish diverse and semantically enriched representations.<n>An extensive series of experiments have been conducted to evaluate the proposed methodology's efficacy.
  arXiv  Detail & Related papers  (2025-04-14T15:22:51Z)
• Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization [20.783312940122297]
  Text-to-image (T2I) diffusion models have achieved remarkable success in generating high-quality images from textual prompts.<n>However, their ability to store vast amounts of knowledge raises concerns in scenarios where selective forgetting is necessary.<n>We propose textbfDynamic Mask coupled with Concept-Aware Loss, a novel unlearning framework designed for multi-concept forgetting.
  arXiv  Detail & Related papers  (2025-04-12T01:38:58Z)
• Language Guided Concept Bottleneck Models for Interpretable Continual Learning [62.09201360376577]
  Continual learning aims to enable learning systems to acquire new knowledge constantly without forgetting previously learned information. Most existing CL methods focus primarily on preserving learned knowledge to improve model performance. We introduce a novel framework that integrates language-guided Concept Bottleneck Models to address both challenges.
  arXiv  Detail & Related papers  (2025-03-30T02:41:55Z)
• CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learning [6.738409533239947]
  Deep neural networks struggle with catastrophic forgetting when learning tasks sequentially. Recent approaches constrain updates to subspaces using gradient projection. We propose Conceptor-based gradient projection for Deep Continual Learning (CODE-CL)
  arXiv  Detail & Related papers  (2024-11-21T22:31:06Z)
• Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning [79.46570165281084]
  We propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks.
  arXiv  Detail & Related papers  (2024-11-11T07:36:19Z)
• Temporal-Difference Variational Continual Learning [89.32940051152782]
  A crucial capability of Machine Learning models in real-world applications is the ability to continuously learn new tasks. In Continual Learning settings, models often struggle to balance learning new tasks with retaining previous knowledge. We propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations.
  arXiv  Detail & Related papers  (2024-10-10T10:58:41Z)
• AdaCBM: An Adaptive Concept Bottleneck Model for Explainable and Accurate Diagnosis [38.16978432272716]
  The integration of vision-language models such as CLIP and Concept Bottleneck Models (CBMs) offers a promising approach to explaining deep neural network (DNN) decisions. While CLIP provides both explainability and zero-shot classification capability, its pre-training on generic image and text data may limit its classification accuracy and applicability to medical image diagnostic tasks. This paper takes an unconventional approach by re-examining the CBM framework through the lens of its geometrical representation as a simple linear classification system.
  arXiv  Detail & Related papers  (2024-08-04T11:59:09Z)
• Editable Concept Bottleneck Models [36.38845338945026]
  Concept Bottleneck Models (CBMs) have garnered much attention for their ability to elucidate the prediction process through a human-understandable concept layer. In many scenarios, we always need to remove/insert some training data or new concepts from trained CBMs due to different reasons, such as privacy concerns, data mislabelling, spurious concepts, and concept annotation errors. We propose Editable Concept Bottleneck Models (ECBMs) to address these challenges. Specifically, ECBMs support three different levels of data removal: concept-label-level, concept-level, and data-level.
  arXiv  Detail & Related papers  (2024-05-24T11:55:46Z)
• ReconBoost: Boosting Can Achieve Modality Reconcilement [89.4377895465204]
  We study the modality-alternating learning paradigm to achieve reconcilement. We propose a new method called ReconBoost to update a fixed modality each time. We show that the proposed method resembles Friedman's Gradient-Boosting (GB) algorithm, where the updated learner can correct errors made by others.
  arXiv  Detail & Related papers  (2024-05-15T13:22:39Z)
• A Unified and General Framework for Continual Learning [58.72671755989431]
  Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
  arXiv  Detail & Related papers  (2024-03-20T02:21:44Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Multi-View Class Incremental Learning [57.14644913531313]
  Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance. This paper investigates a novel paradigm called multi-view class incremental learning (MVCIL), where a single model incrementally classifies new classes from a continual stream of views.
  arXiv  Detail & Related papers  (2023-06-16T08:13:41Z)
• ICICLE: Interpretable Class Incremental Continual Learning [35.105786309067895]
  Interpretable Class-InCremental LEarning (ICICLE) is an exemplar-free approach that adopts a prototypical part-based approach. Our experimental results demonstrate that ICICLE reduces the interpretability concept drift and outperforms the existing exemplar-free methods of common class-incremental learning.
  arXiv  Detail & Related papers  (2023-03-14T11:31:45Z)
• A Memory Transformer Network for Incremental Learning [64.0410375349852]
  We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from. Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes. One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks.
  arXiv  Detail & Related papers  (2022-10-10T08:27:28Z)
• Learning an evolved mixture model for task-free continual learning [11.540150938141034]
  We address the Task-Free Continual Learning (TFCL) in which a model is trained on non-stationary data streams with no explicit task information. We introduce two simple dropout mechanisms to selectively remove stored examples in order to avoid memory overload.
  arXiv  Detail & Related papers  (2022-07-11T16:01:27Z)
• FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation [69.34011200590817]
  We introduce FiLM-Ensemble, a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation. By modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity. We show that FiLM-Ensemble outperforms other implicit ensemble methods, and it comes very close to the upper bound of an explicit ensemble of networks.
  arXiv  Detail & Related papers  (2022-05-31T18:33:15Z)
• Post-hoc Concept Bottleneck Models [11.358495577593441]
  Concept Bottleneck Models (CBMs) map the inputs onto a set of interpretable concepts and use the concepts to make predictions. CBMs are restrictive in practice as they require concept labels in the training data to learn the bottleneck and do not leverage strong pretrained models. We show that we can turn any neural network into a PCBM without sacrificing model performance while still retaining interpretability benefits.
  arXiv  Detail & Related papers  (2022-05-31T00:29:26Z)
• Incremental Object Detection via Meta-Learning [77.55310507917012]
  We propose a meta-learning approach that learns to reshape model gradients, such that information across incremental tasks is optimally shared. In comparison to existing meta-learning methods, our approach is task-agnostic, allows incremental addition of new-classes and scales to high-capacity models for object detection.
  arXiv  Detail & Related papers  (2020-03-17T13:40:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.