SimCon Loss with Multiple Views for Text Supervised Semantic
Segmentation
- URL: http://arxiv.org/abs/2302.03432v1
- Date: Tue, 7 Feb 2023 12:36:35 GMT
- Title: SimCon Loss with Multiple Views for Text Supervised Semantic
Segmentation
- Abstract summary: Learning to segment images purely by relying on the image-text alignment from web data can lead to sub-optimal performance due to noise in the data.
This paper proposes a novel loss function termed SimCon, which accounts for intra-modal similarities to determine the appropriate set of positive samples to align.
- Score: 10.735102459660057
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Learning to segment images purely by relying on the image-text alignment from
web data can lead to sub-optimal performance due to noise in the data. The
noise comes from the samples where the associated text does not correlate with
the image's visual content. Instead of purely relying on the alignment from the
noisy data, this paper proposes a novel loss function termed SimCon, which
accounts for intra-modal similarities to determine the appropriate set of
positive samples to align. Further, using multiple views of the image (created
synthetically) for training and combining the SimCon loss with it makes the
training more robust. This version of the loss is termed MV-SimCon. The
empirical results demonstrate that using the proposed loss function leads to
consistent improvements on zero-shot, text supervised semantic segmentation and
outperforms state-of-the-art by $+3.0\%$, $+3.3\%$ and $+6.9\%$ on PASCAL VOC,
PASCAL Context and MSCOCO, respectively. With test time augmentations, we set a
new record by improving these results further to $58.7\%$, $26.6\%$, and
$33.3\%$ on PASCAL VOC, PASCAL Context, and MSCOCO, respectively. In addition,
using the proposed loss function leads to robust training and faster
convergence.
Related papers
- Robust Remote Sensing Image-Text Retrieval with Noisy Correspondence [24.863417464867776]
We propose a novel Robust Remote Sensing Image-Text Retrieval (RRSITR) paradigm that designs a self-paced learning strategy to mimic human cognitive learning patterns.<n>Experiments on three popular benchmark datasets demonstrate that the proposed RRSITR significantly outperforms the state-of-the-art methods.
arXiv Detail & Related papers (2026-03-30T07:55:07Z) - Dynamic Uncertainty Learning with Noisy Correspondence for Text-Based Person Search [2.3099448395832956]
Large-scale text-image datasets are created from co-occurrences online.<n>Existing methods often focus on negative samples, amplifying the noise.<n>We propose Dynamic Uncertainty and Alignment framework, which includes Key Feature Selector (KFS) and a new loss function, Dynamic Softmax Hinge Loss (DSH-Loss)<n>Our experiments show that the method offers strong noise resistance and improves retrieval performance in both low- and high-noise scenarios.
arXiv Detail & Related papers (2025-05-10T08:35:36Z) - Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning [70.98890307376548]
We propose a novel Patch-wise Cross-modal feature Mix-up (PCM) mechanism to adaptively mitigate the unfaithful contents during training.
Our PCM-Net ranks first in both in-domain and cross-domain zero-shot image captioning.
arXiv Detail & Related papers (2024-12-31T13:39:08Z) - TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations.
We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution.
We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
arXiv Detail & Related papers (2024-11-04T19:24:59Z) - MoTaDual: Modality-Task Dual Alignment for Enhanced Zero-shot Composed Image Retrieval [20.612534837883892]
Composed Image Retrieval (CIR) is a challenging vision-language task, utilizing bi-modal (image+text) queries to retrieve target images.
In this paper, we propose a two-stage framework to tackle both discrepancies.
MoTaDual achieves the state-of-the-art performance across four widely used ZS-CIR benchmarks, while maintaining low training time and computational cost.
arXiv Detail & Related papers (2024-10-31T08:49:05Z) - $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs [62.565573316667276]
We develop an objective that encodes how a sample relates to others.
We train vision models based on similarities in class or text caption descriptions.
Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8%$ on ImageNet and $18.1%$ on ImageNet Real.
arXiv Detail & Related papers (2024-07-25T15:38:16Z) - Text Augmented Spatial-aware Zero-shot Referring Image Segmentation [60.84423786769453]
We introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework.
TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial for mask post-processing.
The proposed method clearly outperforms state-of-the-art zero-shot referring image segmentation methods.
arXiv Detail & Related papers (2023-10-27T10:52:50Z) - ALIP: Adaptive Language-Image Pre-training with Synthetic Caption [78.93535202851278]
Contrastive Language-Image Pre-training (CLIP) has significantly boosted the performance of various vision-language tasks.
The presence of intrinsic noise and unmatched image-text pairs in web data can potentially affect the performance of representation learning.
We propose an Adaptive Language-Image Pre-training (ALIP), a bi-path model that integrates supervision from both raw text and synthetic caption.
arXiv Detail & Related papers (2023-08-16T15:19:52Z) - Improving Multimodal Datasets with Image Captioning [65.74736570293622]
We study how generated captions can increase the utility of web-scraped datapoints with nondescript text.
Our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text.
arXiv Detail & Related papers (2023-07-19T17:47:12Z) - Rethinking Generalization in Few-Shot Classification [28.809141478504532]
Single image-level annotations only correctly describe an often small subset of an image's content.
In this paper, we take a closer look at the implications in the context of $textitfew-shot learning$.
We build on recent advances in unsupervised training of networks via masked image modelling to overcome the lack of fine-grained labels.
arXiv Detail & Related papers (2022-06-15T03:05:21Z) - PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model
Pretraining [68.84339672878066]
We introduce PyramidCLIP, which constructs an input pyramid with different semantic levels, and aligns visual elements and linguistic elements in the form of hierarchy.
Experiments on three downstream tasks, including zero-shot image classification, zero-shot image-text retrieval and image object detection, verify the effectiveness of the proposed PyramidCLIP.
arXiv Detail & Related papers (2022-04-29T13:38:42Z) - USIS: Unsupervised Semantic Image Synthesis [9.613134538472801]
We propose a new Unsupervised paradigm for Semantic Image Synthesis (USIS)
USIS learns to output images with visually separable semantic classes using a self-supervised segmentation loss.
In order to match the color and texture distribution of real images without losing high-frequency information, we propose to use whole image wavelet-based discrimination.
arXiv Detail & Related papers (2021-09-29T20:48:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.