Relieving Triplet Ambiguity: Consensus Network for Language-Guided Image
Retrieval
- URL: http://arxiv.org/abs/2306.02092v1
- Date: Sat, 3 Jun 2023 11:50:44 GMT
- Title: Relieving Triplet Ambiguity: Consensus Network for Language-Guided Image
Retrieval
- Authors: Xu Zhang, Zhedong Zheng, Xiaohan Wang, Yi Yang
- Abstract summary: We propose a novel Consensus Network (Css-Net) that self-adaptively learns from noisy triplets to minimize the negative effects of triplet ambiguity.
Css-Net can alleviate triplet ambiguity, achieving competitive performance on benchmarks, such as $+2.77%$ R@10 and $+6.67%$ R@50 on FashionIQ.
- Score: 48.914550252133125
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Language-guided image retrieval enables users to search for images and
interact with the retrieval system more naturally and expressively by using a
reference image and a relative caption as a query. Most existing studies mainly
focus on designing image-text composition architecture to extract
discriminative visual-linguistic relations. Despite great success, we identify
an inherent problem that obstructs the extraction of discriminative features
and considerably compromises model training: \textbf{triplet ambiguity}. This
problem stems from the annotation process wherein annotators view only one
triplet at a time. As a result, they often describe simple attributes, such as
color, while neglecting fine-grained details like location and style. This
leads to multiple false-negative candidates matching the same modification
text. We propose a novel Consensus Network (Css-Net) that self-adaptively
learns from noisy triplets to minimize the negative effects of triplet
ambiguity. Inspired by the psychological finding that groups perform better
than individuals, Css-Net comprises 1) a consensus module featuring four
distinct compositors that generate diverse fused image-text embeddings and 2) a
Kullback-Leibler divergence loss, which fosters learning among the compositors,
enabling them to reduce biases learned from noisy triplets and reach a
consensus. The decisions from four compositors are weighted during evaluation
to further achieve consensus. Comprehensive experiments on three datasets
demonstrate that Css-Net can alleviate triplet ambiguity, achieving competitive
performance on benchmarks, such as $+2.77\%$ R@10 and $+6.67\%$ R@50 on
FashionIQ.
Related papers
- Noisy-Correspondence Learning for Text-to-Image Person Re-identification [50.07634676709067]
We propose a novel Robust Dual Embedding method (RDE) to learn robust visual-semantic associations even with noisy correspondences.
Our method achieves state-of-the-art results both with and without synthetic noisy correspondences on three datasets.
arXiv Detail & Related papers (2023-08-19T05:34:13Z) - Ranking-aware Uncertainty for Text-guided Image Retrieval [17.70430913227593]
We propose a novel ranking-aware uncertainty approach to model many-to-many correspondences.
Compared to the existing state-of-the-art methods, our proposed method achieves significant results on two public datasets.
arXiv Detail & Related papers (2023-08-16T03:48:19Z) - PV2TEA: Patching Visual Modality to Textual-Established Information
Extraction [59.76117533540496]
We patch the visual modality to the textual-established attribute information extractor.
PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes.
Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
arXiv Detail & Related papers (2023-06-01T05:39:45Z) - CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for
Image-Text Retrieval [108.48540976175457]
We propose Coupled Diversity-Sensitive Momentum Constrastive Learning (CODER) for improving cross-modal representation.
We introduce dynamic dictionaries for both modalities to enlarge the scale of image-text pairs, and diversity-sensitiveness is achieved by adaptive negative pair weighting.
Experiments conducted on two popular benchmarks, i.e. MSCOCO and Flicker30K, validate CODER remarkably outperforms the state-of-the-art approaches.
arXiv Detail & Related papers (2022-08-21T08:37:50Z) - Contrastive Semantic Similarity Learning for Image Captioning Evaluation
with Intrinsic Auto-encoder [52.42057181754076]
Motivated by the auto-encoder mechanism and contrastive representation learning advances, we propose a learning-based metric for image captioning.
We develop three progressive model structures to learn the sentence level representations.
Experiment results show that our proposed method can align well with the scores generated from other contemporary metrics.
arXiv Detail & Related papers (2021-06-29T12:27:05Z) - Learning to Compare Relation: Semantic Alignment for Few-Shot Learning [48.463122399494175]
We present a novel semantic alignment model to compare relations, which is robust to content misalignment.
We conduct extensive experiments on several few-shot learning datasets.
arXiv Detail & Related papers (2020-02-29T08:37:02Z) - Deep Multimodal Image-Text Embeddings for Automatic Cross-Media
Retrieval [0.0]
We introduce an end-to-end deep multimodal convolutional-recurrent network for learning both vision and language representations simultaneously.
The model learns which pairs are a match (positive) and which ones are a mismatch (negative) using a hinge-based triplet ranking.
arXiv Detail & Related papers (2020-02-23T23:58:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.