Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards

• URL: http://arxiv.org/abs/2008.02693v2
• Date: Sat, 23 Apr 2022 18:07:49 GMT
• Title: Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards
• Abstract summary: The goal of this work is to develop a novel learning framework for accurate and expressive fashion captioning. We seed the description of an item by first identifying its attributes, and introduce attribute-level semantic (ALS) reward and sentence-level semantic (SLS) reward as metrics to improve the quality of text descriptions.
• Score: 41.04264950157818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating accurate descriptions for online fashion items is important not only for enhancing customers' shopping experiences, but also for the increase of online sales. Besides the need of correctly presenting the attributes of items, the expressions in an enchanting style could better attract customer interests. The goal of this work is to develop a novel learning framework for accurate and expressive fashion captioning. Different from popular work on image captioning, it is hard to identify and describe the rich attributes of fashion items. We seed the description of an item by first identifying its attributes, and introduce attribute-level semantic (ALS) reward and sentence-level semantic (SLS) reward as metrics to improve the quality of text descriptions. We further integrate the training of our model with maximum likelihood estimation (MLE), attribute embedding, and Reinforcement Learning (RL). To facilitate the learning, we build a new FAshion CAptioning Dataset (FACAD), which contains 993K images and 130K corresponding enchanting and diverse descriptions. Experiments on FACAD demonstrate the effectiveness of our model.

Related papers

• RA-CoA: Training-free Fashion Image Captioning via Retrieval-Augmented Chain-of-Attributes [5.735692008673119]
  Fashion Image Captioning (FIC) plays a vital role in enhancing user experience and product search in e-commerce platforms.<n>We propose RA-CoA, a training-free framework that disentangles fashion image captioning into two interpretable stages.<n> RA-CoA significantly improves caption quality, achieving an average gain of 26.3% METEOR score over zero-shot captioning.
  arXiv  Detail & Related papers  (2026-09-12T18:51:24Z)
• Click it or Leave it: Detecting and Spoiling Clickbait with Informativeness Measures and Large Language Models [1.5658704610960574]
  We present a hybrid approach to clickbait detection that combines transformer-based text embeddings with linguistically motivated informativeness features.<n>Our best-performing model, XGBoost over embeddings augmented with 15 explicit features, achieves an F1-score of 91%.<n>The proposed feature set enhances interpretability by highlighting salient linguistic cues such as second-person pronouns, superlatives, numerals, and attention-oriented punctuation.
  arXiv  Detail & Related papers  (2026-02-20T12:16:08Z)
• Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes [54.50887214639301]
  We propose an innovative approach that harnesses web-crawled descriptions, leveraging a large-language model to extract relevant keywords.<n>This method reduces the need for human annotators and eliminates the laborious manual process of attribute data creation.<n>In our zero-shot experiments, our model achieves accuracies of 81.0%, 53.1%, and 68.9% on UCF-101, HMDB-51, and Kinetics-600, respectively.
  arXiv  Detail & Related papers  (2025-10-31T07:45:44Z)
• LATex: Leveraging Attribute-based Text Knowledge for Aerial-Ground Person Re-Identification [63.07563443280147]
  We propose a novel framework named LATex for AG-ReID. It adopts prompt-tuning strategies to leverage attribute-based text knowledge. Our framework can fully leverage attribute-based text knowledge to improve the AG-ReID.
  arXiv  Detail & Related papers  (2025-03-31T04:47:05Z)
• FashionFAE: Fine-grained Attributes Enhanced Fashion Vision-Language Pre-training [9.149799210311468]
  We propose a novel approach for the fashion domain, which focuses on the detailed characteristics of fashion data. An attribute-emphasized text prediction task is proposed to predict fine-grained attributes of the items. A novel attribute-promoted image reconstruction task is proposed, which further enhances the fine-grained ability of the model.
  arXiv  Detail & Related papers  (2024-12-28T03:45:49Z)
• UnMA-CapSumT: Unified and Multi-Head Attention-driven Caption Summarization Transformer [6.351779356923131]
  This paper presents a novel Unified Attention and Multi-Head Attention-driven Caption Summarization Transformer (UnMA-CapSumT) based Captioning Framework. It utilizes both factual captions and stylized captions generated by the Modified Adaptive Attention-based factual image captioning model (MAA-FIC) and Style Factored Bi-LSTM with attention (SF-Bi-ALSTM) driven stylized image captioning model respectively.
  arXiv  Detail & Related papers  (2024-12-16T14:57:40Z)
• CLIP-SCGI: Synthesized Caption-Guided Inversion for Person Re-Identification [9.996589403019675]
  Person re-identification (ReID) has recently benefited from large pretrained vision-language models such as Contrastive Language-Image Pre-Training (CLIP) We propose one straightforward solution by leveraging existing image captioning models to generate pseudo captions for person images. We introduce CLIP-SCGI, a framework that leverages synthesized captions to guide the learning of discriminative and robust representations.
  arXiv  Detail & Related papers  (2024-10-12T06:24:33Z)
• ITEm: Unsupervised Image-Text Embedding Learning for eCommerce [9.307841602452678]
  Product embedding serves as a cornerstone for a wide range of applications in eCommerce. We present an image-text embedding model (ITEm) that is designed to better attend to image and text modalities. We evaluate the pre-trained ITEm on two tasks: the search for extremely similar products and the prediction of product categories.
  arXiv  Detail & Related papers  (2023-10-22T15:39:44Z)
• Exploring Fine-Grained Representation and Recomposition for Cloth-Changing Person Re-Identification [78.52704557647438]
  We propose a novel FIne-grained Representation and Recomposition (FIRe$2$) framework to tackle both limitations without any auxiliary annotation or data. Experiments demonstrate that FIRe$2$ can achieve state-of-the-art performance on five widely-used cloth-changing person Re-ID benchmarks.
  arXiv  Detail & Related papers  (2023-08-21T12:59:48Z)
• Learning Conditional Attributes for Compositional Zero-Shot Learning [78.24309446833398]
  Compositional Zero-Shot Learning (CZSL) aims to train models to recognize novel compositional concepts. One of the challenges is to model attributes interacted with different objects, e.g., the attribute wet" in wet apple" and wet cat" is different. We argue that attributes are conditioned on the recognized object and input image and explore learning conditional attribute embeddings.
  arXiv  Detail & Related papers  (2023-05-29T08:04:05Z)
• Attribute-aware Explainable Complementary Clothing Recommendation [37.30129304097086]
  This work aims to tackle the explainability challenge in fashion recommendation tasks by proposing a novel Attribute-aware Fashion Recommender (AFRec) AFRec recommender assesses the outfit compatibility by explicitly leveraging the extracted attribute-level representations from each item's visual feature. The attributes serve as the bridge between two fashion items, where we quantify the affinity of a pair of items through the learned compatibility between their attributes.
  arXiv  Detail & Related papers  (2021-07-04T14:56:07Z)
• PAM: Understanding Product Images in Cross Product Category Attribute Extraction [40.332066960433245]
  This work proposes a more inclusive framework that fully utilizes different modalities for attribute extraction. Inspired by recent works in visual question answering, we use a transformer based sequence to sequence model to fuse representations of product text, Optical Character Recognition (OCR) tokens and visual objects detected in the product image. The framework is further extended with the capability to extract attribute value across multiple product categories with a single model.
  arXiv  Detail & Related papers  (2021-06-08T18:30:17Z)
• Rich Semantics Improve Few-shot Learning [49.11659525563236]
  We show that by using 'class-level' language descriptions, that can be acquired with minimal annotation cost, we can improve the few-shot learning performance. We develop a Transformer based forward and backward encoding mechanism to relate visual and semantic tokens.
  arXiv  Detail & Related papers  (2021-04-26T16:48:27Z)
• Attr2Style: A Transfer Learning Approach for Inferring Fashion Styles via Apparel Attributes [7.330801562698453]
  We propose a transfer-learning based image captioning model that is trained on a source dataset with sufficient attribute-based ground-truth captions. We leverage this fact and train our model in an encoder-decoder based framework using attention mechanism. A Proof Of Concept for our model is under pilot at Myntra where it is exposed to some internal users for feedback.
  arXiv  Detail & Related papers  (2020-08-26T16:42:21Z)
• Automatic Validation of Textual Attribute Values in E-commerce Catalog by Learning with Limited Labeled Data [61.789797281676606]
  We propose a novel meta-learning latent variable approach, called MetaBridge. It can learn transferable knowledge from a subset of categories with limited labeled data. It can capture the uncertainty of never-seen categories with unlabeled data.
  arXiv  Detail & Related papers  (2020-06-15T21:31:05Z)
• Fine-Grained Fashion Similarity Learning by Attribute-Specific Embedding Network [59.479783847922135]
  We propose an Attribute-Specific Embedding Network (ASEN) to jointly learn multiple attribute-specific embeddings in an end-to-end manner. ASEN is able to locate the related regions and capture the essential patterns under the guidance of the specified attribute. Experiments on four fashion-related datasets show the effectiveness of ASEN for fine-grained fashion similarity learning.
  arXiv  Detail & Related papers  (2020-02-07T14:42:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.