Heterogeneous Semantic Transfer for Multi-label Recognition with Partial Labels

• URL: http://arxiv.org/abs/2205.11131v4
• Date: Mon, 15 Jul 2024 14:32:29 GMT
• Title: Heterogeneous Semantic Transfer for Multi-label Recognition with Partial Labels
• Authors: Tianshui Chen, Tao Pu, Lingbo Liu, Yukai Shi, Zhijing Yang, Liang Lin,
• Abstract summary: Multi-label image recognition with partial labels (MLR-PL) may greatly reduce the cost of annotation and thus facilitate large-scale MLR. We find that strong semantic correlations exist within each image and across different images. These correlations can help transfer the knowledge possessed by the known labels to retrieve the unknown labels.
• Score: 70.45813147115126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-label image recognition with partial labels (MLR-PL), in which some labels are known while others are unknown for each image, may greatly reduce the cost of annotation and thus facilitate large-scale MLR. We find that strong semantic correlations exist within each image and across different images, and these correlations can help transfer the knowledge possessed by the known labels to retrieve the unknown labels and thus improve the performance of the MLR-PL task (see Figure 1). In this work, we propose a novel heterogeneous semantic transfer (HST) framework that consists of two complementary transfer modules that explore both within-image and cross-image semantic correlations to transfer the knowledge possessed by known labels to generate pseudo labels for the unknown labels. Specifically, an intra-image semantic transfer (IST) module learns an image-specific label co-occurrence matrix for each image and maps the known labels to complement the unknown labels based on these matrices. Additionally, a cross-image transfer (CST) module learns category-specific feature-prototype similarities and then helps complement the unknown labels that have high degrees of similarity with the corresponding prototypes. Finally, both the known and generated pseudo labels are used to train MLR models. Extensive experiments conducted on the Microsoft COCO, Visual Genome, and Pascal VOC 2007 datasets show that the proposed HST framework achieves superior performance to that of current state-of-the-art algorithms. Specifically, it obtains mean average precision (mAP) improvements of 1.4%, 3.3%, and 0.4% on the three datasets over the results of the best-performing previously developed algorithm.

Related papers

• Dual-Perspective Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels [70.36722026729859]
  We propose a dual-perspective semantic-aware representation blending (DSRB) that blends multi-granularity category-specific semantic representation across different images. The proposed DS consistently outperforms current state-of-the-art algorithms on all proportion label settings.
  arXiv  Detail & Related papers  (2022-05-26T00:33:44Z)
• Graph Attention Transformer Network for Multi-Label Image Classification [50.0297353509294]
  We propose a general framework for multi-label image classification that can effectively mine complex inter-label relationships. Our proposed methods can achieve state-of-the-art performance on three datasets.
  arXiv  Detail & Related papers  (2022-03-08T12:39:05Z)
• Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels [86.17081952197788]
  We propose to blend category-specific representation across different images to transfer information of known labels to complement unknown labels. Experiments on the MS-COCO, Visual Genome, Pascal VOC 2007 datasets show that the proposed SARB framework obtains superior performance over current leading competitors.
  arXiv  Detail & Related papers  (2022-03-04T07:56:16Z)
• Structured Semantic Transfer for Multi-Label Recognition with Partial Labels [85.6967666661044]
  We propose a structured semantic transfer (SST) framework that enables training multi-label recognition models with partial labels. The framework consists of two complementary transfer modules that explore within-image and cross-image semantic correlations. Experiments on the Microsoft COCO, Visual Genome and Pascal VOC datasets show that the proposed SST framework obtains superior performance over current state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-12-21T02:15:01Z)
• Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition [75.44233392355711]
  KGGR framework exploits prior knowledge of statistical label correlations with deep neural networks. It first builds a structured knowledge graph to correlate different labels based on statistical label co-occurrence. Then, it introduces the label semantics to guide learning semantic-specific features. It exploits a graph propagation network to explore graph node interactions.
  arXiv  Detail & Related papers  (2020-09-20T15:05:29Z)
• SSKD: Self-Supervised Knowledge Distillation for Cross Domain Adaptive Person Re-Identification [25.96221714337815]
  Domain adaptive person re-identification (re-ID) is a challenging task due to the large discrepancy between the source domain and the target domain. Existing methods mainly attempt to generate pseudo labels for unlabeled target images by clustering algorithms. We propose a Self-Supervised Knowledge Distillation (SSKD) technique containing two modules, the identity learning and the soft label learning.
  arXiv  Detail & Related papers  (2020-09-13T10:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.