Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning

• URL: http://arxiv.org/abs/2511.15633v1
• Date: Wed, 19 Nov 2025 17:14:47 GMT
• Title: Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning
• Authors: Tao Hu, Lan Li, Zhen-Hao Xie, Da-Wei Zhou,
• Abstract summary: Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge.<n>But real-world visual and linguistic concepts are inherently hierarchical.<n>We propose HASTEN that anchors hierarchical information into CIL to reduce catastrophic forgetting.
• Score: 11.82771798674077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge. Recently, vision-language models like CLIP offer transferable features via multi-modal pre-training, making them well-suited for CIL. However, real-world visual and linguistic concepts are inherently hierarchical: a textual concept like "dog" subsumes fine-grained categories such as "Labrador" and "Golden Retriever," and each category entails its images. But existing CLIP-based CIL methods fail to explicitly capture this inherent hierarchy, leading to fine-grained class features drift during incremental updates and ultimately to catastrophic forgetting. To address this challenge, we propose HASTEN (Hierarchical Semantic Tree Anchoring) that anchors hierarchical information into CIL to reduce catastrophic forgetting. First, we employ an external knowledge graph as supervision to embed visual and textual features in hyperbolic space, effectively preserving hierarchical structure as data evolves. Second, to mitigate catastrophic forgetting, we project gradients onto the null space of the shared hyperbolic mapper, preventing interference with prior tasks. These two steps work synergistically to enable the model to resist forgetting by maintaining hierarchical relationships. Extensive experiments show that HASTEN consistently outperforms existing methods while providing a unified structured representation.

Related papers

• Unsupervised Hierarchical Skill Discovery [11.230382111014073]
  We consider the problem of unsupervised skill segmentation and hierarchical structure discovery in reinforcement learning.<n>We propose a method that segments unlabelled trajectories into skills and induces a hierarchical structure over them using a grammar-based approach.<n>We evaluate our approach in high-dimensional, pixel-based environments, including Craftax and the full, unmodified version of Minecraft.
  arXiv  Detail & Related papers  (2026-01-30T16:41:13Z)
• Hierarchical Representation Matching for CLIP-based Class-Incremental Learning [80.2317078787969]
  Class-Incremental Learning (CIL) aims to endow models with the ability to continuously adapt to evolving data streams.<n>Recent advances in pre-trained vision-language models (e.g., CLIP) provide a powerful foundation for this task.<n>We introduce HiErarchical Representation MAtchiNg (HERMAN) for CLIP-based CIL.
  arXiv  Detail & Related papers  (2025-09-26T17:59:51Z)
• Learning and Evaluating Hierarchical Feature Representations [3.770103075126785]
  We propose a novel framework, Hierarchical Composition of Orthogonal Subspaces (Hier-COS)<n>Hier-COS learns to map deep feature embeddings into a vector space that is, by design, consistent with the structure of a given taxonomy tree.<n>We demonstrate that Hier-COS achieves state-of-the-art hierarchical performance across all the datasets while simultaneously beating top-1 accuracy in all but one case.
  arXiv  Detail & Related papers  (2025-03-10T20:59:41Z)
• HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding [18.95003393925676]
  When classifying categories at different hierarchy levels, traditional uni-modal approaches focus primarily on image features, revealing limitations in complex scenarios.<n>Recent studies integrating Vision-Language Models (VLMs) with class hierarchies have shown promise, yet they fall short of fully exploiting the hierarchical relationships.<n>We propose a novel framework that effectively combines CLIP with a deeper exploitation of the Hierarchical class structure via Graph representation learning.
  arXiv  Detail & Related papers  (2023-11-23T15:42:42Z)
• Online Continual Learning on Hierarchical Label Expansion [28.171890301966616]
  We propose a novel multi-level hierarchical class incremental task configuration with an online learning constraint, called hierarchical label expansion (HLE) Our configuration allows a network to first learn coarse-grained classes, with data labels continually expanding to more fine-grained classes in various hierarchy depths. Our experiments demonstrate that our proposed method can effectively use hierarchy on our HLE setup to improve classification accuracy across all levels of hierarchies.
  arXiv  Detail & Related papers  (2023-08-28T07:42:26Z)
• Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression [59.97965005675144]
  Contrastive learning (CL) has emerged as a powerful technique for representation learning, with or without label supervision. We provide the first unified theoretically rigorous framework to determine textitwhich features are learnt by CL. We present increasing embedding dimensionality and improving the quality of data augmentations as two theoretically motivated solutions.
  arXiv  Detail & Related papers  (2023-05-25T23:37:22Z)
• AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learning [53.32576252950481]
  Continual learning aims to enable a model to incrementally learn knowledge from sequentially arrived data. In this paper, we propose a non-incremental learner, named AttriCLIP, to incrementally extract knowledge of new classes or tasks.
  arXiv  Detail & Related papers  (2023-05-19T07:39:17Z)
• Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework [75.79736930414715]
  We present a hierarchical multi-label representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. We introduce novel hierarchy preserving losses, which jointly apply a hierarchical penalty to the contrastive loss, and enforce the hierarchy constraint.
  arXiv  Detail & Related papers  (2022-04-27T21:41:44Z)
• Deep Hierarchical Semantic Segmentation [76.40565872257709]
  hierarchical semantic segmentation (HSS) aims at structured, pixel-wise description of visual observation in terms of a class hierarchy. HSSN casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. With hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space, so as to generate well-structured pixel representations.
  arXiv  Detail & Related papers  (2022-03-27T15:47:44Z)
• Self-Supervised Class Incremental Learning [51.62542103481908]
  Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels. When updating them based on the new class data, they suffer from catastrophic forgetting: the model cannot discern old class data clearly from the new. In this paper, we explore the performance of Self-Supervised representation learning in Class Incremental Learning (SSCIL) for the first time.
  arXiv  Detail & Related papers  (2021-11-18T06:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.