Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning

• URL: http://arxiv.org/abs/2408.12253v2
• Date: Sun, 25 Aug 2024 14:13:40 GMT
• Title: Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning
• Authors: Ziming Liu, Jingcai Guo, Song Guo, Xiaocheng Lu,
• Abstract summary: This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL) It is trained to recognize multiple unseen classes within a sample based on seen classes and auxiliary knowledge. We propose a novel and comprehensive visual-semantic framework for MLZSL, dubbed Epsilon, to fully make use of such properties.
• Score: 23.96220607033524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL), wherein the model is trained to recognize multiple unseen classes within a sample (e.g., an image) based on seen classes and auxiliary knowledge, e.g., semantic information. Existing methods usually resort to analyzing the relationship of various seen classes residing in a sample from the dimension of spatial or semantic characteristics and transferring the learned model to unseen ones. However, they neglect the integrity of local and global features. Although the use of the attention structure will accurately locate local features, especially objects, it will significantly lose its integrity, and the relationship between classes will also be affected. Rough processing of global features will also directly affect comprehensiveness. This neglect will make the model lose its grasp of the main components of the image. Relying only on the local existence of seen classes during the inference stage introduces unavoidable bias. In this paper, we propose a novel and comprehensive visual-semantic framework for MLZSL, dubbed Epsilon, to fully make use of such properties and enable a more accurate and robust visual-semantic projection. In terms of spatial information, we achieve effective refinement by group aggregating image features into several semantic prompts. It can aggregate semantic information rather than class information, preserving the correlation between semantics. In terms of global semantics, we use global forward propagation to collect as much information as possible to ensure that semantics are not omitted. Experiments on large-scale MLZSL benchmark datasets NUS-Wide and Open-Images-v4 demonstrate that the proposed Epsilon outperforms other state-of-the-art methods with large margins.

Related papers

• GBE-MLZSL: A Group Bi-Enhancement Framework for Multi-Label Zero-Shot Learning [24.075034737719776]
  This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL) We propose a novel and effective group bi-enhancement framework for MLZSL, dubbed GBE-MLZSL, to fully make use of such properties and enable a more accurate and robust visual-semantic projection. Experiments on large-scale MLZSL benchmark datasets NUS-WIDE and Open-Images-v4 demonstrate that the proposed GBE-MLZSL outperforms other state-of-the-art methods with large margins.
  arXiv  Detail & Related papers  (2023-09-02T12:07:21Z)
• Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation [68.13453771001522]
  We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings. We conduct extensive experiments and evaluate our model on large-scale real-world data.
  arXiv  Detail & Related papers  (2023-06-14T13:07:48Z)
• Cross-modal Representation Learning for Zero-shot Action Recognition [67.57406812235767]
  We present a cross-modal Transformer-based framework, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR) Our model employs a conceptually new pipeline by which visual representations are learned in conjunction with visual-semantic associations in an end-to-end manner. Experiment results show our model considerably improves upon the state of the arts in ZSAR, reaching encouraging top-1 accuracy on UCF101, HMDB51, and ActivityNet benchmark datasets.
  arXiv  Detail & Related papers  (2022-05-03T17:39:27Z)
• Discriminative Region-based Multi-Label Zero-Shot Learning [145.0952336375342]
  Multi-label zero-shot learning (ZSL) is a more realistic counter-part of standard single-label ZSL. We propose an alternate approach towards region-based discriminability-preserving ZSL.
  arXiv  Detail & Related papers  (2021-08-20T17:56:47Z)
• Learning Robust Visual-semantic Mapping for Zero-shot Learning [8.299945169799795]
  We focus on fully empowering the semantic feature space, which is one of the key building blocks of Zero-shot learning (ZSL) In ZSL, the common practice is to train a mapping function between the visual and semantic feature spaces with labeled seen class examples. Under such a paradigm, the ZSL models may easily suffer from the domain shift problem when constructing and reusing the mapping function.
  arXiv  Detail & Related papers  (2021-04-12T17:39:38Z)
• Goal-Oriented Gaze Estimation for Zero-Shot Learning [62.52340838817908]
  We introduce a novel goal-oriented gaze estimation module (GEM) to improve the discriminative attribute localization. We aim to predict the actual human gaze location to get the visual attention regions for recognizing a novel object guided by attribute description. This work implies the promising benefits of collecting human gaze dataset and automatic gaze estimation algorithms on high-level computer vision tasks.
  arXiv  Detail & Related papers  (2021-03-05T02:14:57Z)
• Semantic Disentangling Generalized Zero-Shot Learning [50.259058462272435]
  Generalized Zero-Shot Learning (GZSL) aims to recognize images from both seen and unseen categories. In this paper, we propose a novel feature disentangling approach based on an encoder-decoder architecture. The proposed model aims to distill quality semantic-consistent representations that capture intrinsic features of seen images.
  arXiv  Detail & Related papers  (2021-01-20T05:46:21Z)
• Information Bottleneck Constrained Latent Bidirectional Embedding for Zero-Shot Learning [59.58381904522967]
  We propose a novel embedding based generative model with a tight visual-semantic coupling constraint. We learn a unified latent space that calibrates the embedded parametric distributions of both visual and semantic spaces. Our method can be easily extended to transductive ZSL setting by generating labels for unseen images.
  arXiv  Detail & Related papers  (2020-09-16T03:54:12Z)
• Generative Model-driven Structure Aligning Discriminative Embeddings for Transductive Zero-shot Learning [21.181715602603436]
  We propose a neural network-based model for learning a projection function which aligns the visual and semantic data in the latent space. We show superior performance on standard benchmark datasets AWA1, AWA2, CUB, SUN, FLO, and. We also show the efficacy of our model in the case of extremely less labelled data regime.
  arXiv  Detail & Related papers  (2020-05-09T18:48:20Z)
• A Novel Perspective to Zero-shot Learning: Towards an Alignment of Manifold Structures via Semantic Feature Expansion [17.48923061278128]
  A common practice in zero-shot learning is to train a projection between the visual and semantic feature spaces with labeled seen classes examples. Under such a paradigm, most existing methods easily suffer from the domain shift problem and weaken the performance of zero-shot recognition. We propose a novel model called AMS-SFE that considers the alignment of manifold structures by semantic feature expansion.
  arXiv  Detail & Related papers  (2020-04-30T14:08:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.