Category-Prompt Refined Feature Learning for Long-Tailed Multi-Label Image Classification

• URL: http://arxiv.org/abs/2408.08125v1
• Date: Thu, 15 Aug 2024 12:51:57 GMT
• Title: Category-Prompt Refined Feature Learning for Long-Tailed Multi-Label Image Classification
• Authors: Jiexuan Yan, Sheng Huang, Nankun Mu, Luwen Huangfu, Bo Liu,
• Abstract summary: Category-Prompt Refined Feature Learning (CPRFL) is a novel approach for Long-Tailed Multi-Label image Classification. CPRFL initializes category-prompts from the pretrained CLIP's embeddings and decouples category-specific visual representations. We validate the effectiveness of our method on two LTMLC benchmarks and extensive experiments demonstrate the superiority of our work over baselines.
• Score: 8.139529179222844
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Real-world data consistently exhibits a long-tailed distribution, often spanning multiple categories. This complexity underscores the challenge of content comprehension, particularly in scenarios requiring Long-Tailed Multi-Label image Classification (LTMLC). In such contexts, imbalanced data distribution and multi-object recognition pose significant hurdles. To address this issue, we propose a novel and effective approach for LTMLC, termed Category-Prompt Refined Feature Learning (CPRFL), utilizing semantic correlations between different categories and decoupling category-specific visual representations for each category. Specifically, CPRFL initializes category-prompts from the pretrained CLIP's embeddings and decouples category-specific visual representations through interaction with visual features, thereby facilitating the establishment of semantic correlations between the head and tail classes. To mitigate the visual-semantic domain bias, we design a progressive Dual-Path Back-Propagation mechanism to refine the prompts by progressively incorporating context-related visual information into prompts. Simultaneously, the refinement process facilitates the progressive purification of the category-specific visual representations under the guidance of the refined prompts. Furthermore, taking into account the negative-positive sample imbalance, we adopt the Asymmetric Loss as our optimization objective to suppress negative samples across all classes and potentially enhance the head-to-tail recognition performance. We validate the effectiveness of our method on two LTMLC benchmarks and extensive experiments demonstrate the superiority of our work over baselines. The code is available at https://github.com/jiexuanyan/CPRFL.

Related papers

• Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels [19.740929527669483]
  Multi-label recognition with partial labels (MLR-PL) is a practical task in computer vision. We introduce a semantic decoupling module and a category-specific prompt optimization method in CLIP-based framework. Our method effectively separates information from different categories and achieves better performance compared to CLIP-based baseline method.
  arXiv  Detail & Related papers  (2024-12-14T14:31:36Z)
• Category-Adaptive Cross-Modal Semantic Refinement and Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  We propose a novel category-adaptive cross-modal semantic refinement and transfer (C$2$SRT) framework to explore the semantic correlation. The proposed framework consists of two complementary modules, i.e., intra-category semantic refinement (ISR) module and inter-category semantic transfer (IST) module. Experiments on OV-MLR benchmarks clearly demonstrate that the proposed C$2$SRT framework outperforms current state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-12-09T04:00:18Z)
• Dual-Modal Prompting for Sketch-Based Image Retrieval [76.12076969949062]
  We propose a dual-modal CLIP (DP-CLIP) network, in which an adaptive prompting strategy is designed. We employ a set of images within the target category and the textual category label to respectively construct a set of category-adaptive prompt tokens and channel scales. Our DP-CLIP outperforms the state-of-the-art fine-grained zero-shot method by 7.3% in Acc.@1 on the Sketchy dataset.
  arXiv  Detail & Related papers  (2024-04-29T13:43:49Z)
• Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction [17.989559761931435]
  We propose a novel "Fine-grained Visual-Semantic Interaction" framework for WSI classification. It is designed to enhance the model's generalizability by leveraging the interaction between localized visual patterns and fine-grained pathological semantics. Our method demonstrates robust generalizability and strong transferability, dominantly outperforming the counterparts on the TCGA Lung Cancer dataset.
  arXiv  Detail & Related papers  (2024-02-29T16:29:53Z)
• Balanced Classification: A Unified Framework for Long-Tailed Object Detection [74.94216414011326]
  Conventional detectors suffer from performance degradation when dealing with long-tailed data due to a classification bias towards the majority head categories. We introduce a unified framework called BAlanced CLassification (BACL), which enables adaptive rectification of inequalities caused by disparities in category distribution. BACL consistently achieves performance improvements across various datasets with different backbones and architectures.
  arXiv  Detail & Related papers  (2023-08-04T09:11:07Z)
• LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition [12.62835357920401]
  We propose a unified framework for LTML, namely prompt tuning with class-specific embedding loss (LMPT) Our method significantly surpasses the previous state-of-the-art methods and zero-shot CLIP in LTML.
  arXiv  Detail & Related papers  (2023-05-08T08:14:46Z)
• Not All Instances Contribute Equally: Instance-adaptive Class Representation Learning for Few-Shot Visual Recognition [94.04041301504567]
  Few-shot visual recognition refers to recognize novel visual concepts from a few labeled instances. We propose a novel metric-based meta-learning framework termed instance-adaptive class representation learning network (ICRL-Net) for few-shot visual recognition.
  arXiv  Detail & Related papers  (2022-09-07T10:00:18Z)
• Semantic Representation and Dependency Learning for Multi-Label Image Recognition [76.52120002993728]
  We propose a novel and effective semantic representation and dependency learning (SRDL) framework to learn category-specific semantic representation for each category. Specifically, we design a category-specific attentional regions (CAR) module to generate channel/spatial-wise attention matrices to guide model. We also design an object erasing (OE) module to implicitly learn semantic dependency among categories by erasing semantic-aware regions.
  arXiv  Detail & Related papers  (2022-04-08T00:55:15Z)
• Learning Debiased and Disentangled Representations for Semantic Segmentation [52.35766945827972]
  We propose a model-agnostic and training scheme for semantic segmentation. By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes. Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-10-31T16:15:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.