Addressing Discrepancies in Semantic and Visual Alignment in Neural Networks

• URL: http://arxiv.org/abs/2306.01148v1
• Date: Thu, 1 Jun 2023 21:03:06 GMT
• Title: Addressing Discrepancies in Semantic and Visual Alignment in Neural Networks
• Authors: Natalie Abreu, Nathan Vaska, Victoria Helus
• Abstract summary: We consider the problem of when semantically similar classes are visually dissimilar, and when visual similarity is present among non-similar classes. We propose a data augmentation technique with the goal of better aligning semantically similar classes with arbitrary (non-visual) semantic relationships. Results demonstrate that there is an increase in alignment of semantically similar classes when using our proposed data augmentation method.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For the task of image classification, neural networks primarily rely on visual patterns. In robust networks, we would expect for visually similar classes to be represented similarly. We consider the problem of when semantically similar classes are visually dissimilar, and when visual similarity is present among non-similar classes. We propose a data augmentation technique with the goal of better aligning semantically similar classes with arbitrary (non-visual) semantic relationships. We leverage recent work in diffusion-based semantic mixing to generate semantic hybrids of two classes, and these hybrids are added to the training set as augmented data. We evaluate whether the method increases semantic alignment by evaluating model performance on adversarially perturbed data, with the idea that it should be easier for an adversary to switch one class to a similarly represented class. Results demonstrate that there is an increase in alignment of semantically similar classes when using our proposed data augmentation method.

Related papers

• Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation [13.001629605405954]
  We study universal zero-shot segmentation in this work to achieve panoptic, instance, and semantic segmentation for novel categories without any training samples. We introduce a generative model to synthesize features for unseen categories, which links semantic and visual spaces. The proposed approach achieves impressively state-of-the-art performance on zero-shot panoptic segmentation, instance segmentation, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-19T17:59:16Z)
• Advancing Incremental Few-shot Semantic Segmentation via Semantic-guided Relation Alignment and Adaptation [98.51938442785179]
  Incremental few-shot semantic segmentation aims to incrementally extend a semantic segmentation model to novel classes. This task faces a severe semantic-aliasing issue between base and novel classes due to data imbalance. We propose the Semantic-guided Relation Alignment and Adaptation (SRAA) method that fully considers the guidance of prior semantic information.
  arXiv  Detail & Related papers  (2023-05-18T10:40:52Z)
• Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning [74.48337375174297]
  Generalized Zero-Shot Learning (GZSL) identifies unseen categories by knowledge transferred from the seen domain. We deploy the dual semantic-visual transformer module (DSVTM) to progressively model the correspondences between prototypes and visual features. DSVTM devises an instance-motivated semantic encoder that learns instance-centric prototypes to adapt to different images, enabling the recast of the unmatched semantic-visual pair into the matched one.
  arXiv  Detail & Related papers  (2023-03-27T15:21:43Z)
• Spatial Cross-Attention Improves Self-Supervised Visual Representation Learning [5.085461418671174]
  We introduce an add-on module to facilitate the injection of the knowledge accounting for spatial cross correlations among the samples. This in turn results in distilling intra-class information including feature level locations and cross similarities between same-class instances.
  arXiv  Detail & Related papers  (2022-06-07T21:14:52Z)
• SimMatch: Semi-supervised Learning with Similarity Matching [43.61802702362675]
  SimMatch is a new semi-supervised learning framework that considers semantic similarity and instance similarity. With 400 epochs of training, SimMatch achieves 67.2%, and 74.4% Top-1 Accuracy with 1% and 10% labeled examples on ImageNet.
  arXiv  Detail & Related papers  (2022-03-14T08:08:48Z)
• Cluster-based Contrastive Disentangling for Generalized Zero-Shot Learning [25.92340532509084]
  Generalized Zero-Shot Learning (GZSL) aims to recognize both seen and unseen classes by training only the seen classes. We propose a Cluster-based Contrastive Disentangling (CCD) method to improve GZSL by alleviating the semantic gap and domain shift problems.
  arXiv  Detail & Related papers  (2022-03-05T02:50:12Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Graph Contrastive Clustering [131.67881457114316]
  We propose a novel graph contrastive learning framework, which is then applied to the clustering task and we come up with the Graph Constrastive Clustering(GCC) method. Specifically, on the one hand, the graph Laplacian based contrastive loss is proposed to learn more discriminative and clustering-friendly features. On the other hand, a novel graph-based contrastive learning strategy is proposed to learn more compact clustering assignments.
  arXiv  Detail & Related papers  (2021-04-03T15:32:49Z)
• Self-Supervised Classification Network [3.8073142980733]
  Self-supervised end-to-end classification neural network learns labels and representations simultaneously. First unsupervised end-to-end classification network to perform well on the large-scale ImageNet dataset.
  arXiv  Detail & Related papers  (2021-03-19T19:29:42Z)
• Learning and Evaluating Representations for Deep One-class Classification [59.095144932794646]
  We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T23:33:41Z)
• Adaptive Semantic-Visual Tree for Hierarchical Embeddings [67.01307058209709]
  We propose a hierarchical adaptive semantic-visual tree to depict the architecture of merchandise categories. The tree evaluates semantic similarities between different semantic levels and visual similarities within the same semantic class simultaneously. At each level, we set different margins based on the semantic hierarchy and incorporate them as prior information to learn a fine-grained feature embedding.
  arXiv  Detail & Related papers  (2020-03-08T03:36:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.