FashionFAE: Fine-grained Attributes Enhanced Fashion Vision-Language Pre-training

• URL: http://arxiv.org/abs/2412.19997v2
• Date: Sun, 12 Jan 2025 07:27:03 GMT
• Title: FashionFAE: Fine-grained Attributes Enhanced Fashion Vision-Language Pre-training
• Abstract summary: We propose a novel approach for the fashion domain, which focuses on the detailed characteristics of fashion data.<n>An attribute-emphasized text prediction task is proposed to predict fine-grained attributes of the items.<n>A novel attribute-promoted image reconstruction task is proposed, which further enhances the fine-grained ability of the model.
• Score: 9.149799210311468
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large-scale Vision-Language Pre-training (VLP) has demonstrated remarkable success in the general domain. However, in the fashion domain, items are distinguished by fine-grained attributes like texture and material, which are crucial for tasks such as retrieval. Existing models often fail to leverage these fine-grained attributes from both text and image modalities. To address the above issues, we propose a novel approach for the fashion domain, Fine-grained Attributes Enhanced VLP (FashionFAE), which focuses on the detailed characteristics of fashion data. An attribute-emphasized text prediction task is proposed to predict fine-grained attributes of the items. This forces the model to focus on the salient attributes from the text modality. Additionally, a novel attribute-promoted image reconstruction task is proposed, which further enhances the fine-grained ability of the model by leveraging the representative attributes from the image modality. Extensive experiments show that FashionFAE significantly outperforms State-Of-The-Art (SOTA) methods, achieving 2.9% and 5.2% improvements in retrieval on sub-test and full test sets, respectively, and a 1.6% average improvement in recognition tasks.

Related papers

• RA-CoA: Training-free Fashion Image Captioning via Retrieval-Augmented Chain-of-Attributes [5.735692008673119]
  Fashion Image Captioning (FIC) plays a vital role in enhancing user experience and product search in e-commerce platforms.<n>We propose RA-CoA, a training-free framework that disentangles fashion image captioning into two interpretable stages.<n> RA-CoA significantly improves caption quality, achieving an average gain of 26.3% METEOR score over zero-shot captioning.
  arXiv  Detail & Related papers  (2026-09-12T18:51:24Z)
• Harnessing the Power of Foundation Models for Accurate Material Classification [0.22940141855172033]
  Material classification has emerged as a critical task in computer vision and graphics.<n>Recent advances in vision-language foundation models offer promising avenues to address these issues.<n>We propose a novel framework that effectively harnesses foundation models to overcome data limitations and enhance classification accuracy.
  arXiv  Detail & Related papers  (2026-03-18T06:14:00Z)
• Compositional Attribute Imbalance in Vision Datasets [7.018788111043557]
  We introduce a CLIP-based framework to construct a visual attribute dictionary, enabling automatic evaluation of image attributes.<n>By analyzing both single-attribute imbalance and compositional attribute imbalance, we reveal how the rarity of attributes affects model performance.<n>Our research highlights the importance of modeling visual attribute distributions and provides a scalable solution for long-tail image classification tasks.
  arXiv  Detail & Related papers  (2025-06-17T11:28:07Z)
• LATex: Leveraging Attribute-based Text Knowledge for Aerial-Ground Person Re-Identification [63.07563443280147]
  We propose a novel framework named LATex for AG-ReID. It adopts prompt-tuning strategies to leverage attribute-based text knowledge. Our framework can fully leverage attribute-based text knowledge to improve the AG-ReID.
  arXiv  Detail & Related papers  (2025-03-31T04:47:05Z)
• FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models [112.94440113631897]
  Current methods attempt to distill identity and style from source images.<n>"style" is a broad concept that includes texture, color, and artistic elements, but does not cover other important attributes such as lighting and dynamics.<n>We formulate a more effective approach to decompose the aesthetics of a picture into specific visual attributes, allowing users to apply characteristics such as lighting, texture, and dynamics from different images.
  arXiv  Detail & Related papers  (2024-12-10T17:02:58Z)
• ARMADA: Attribute-Based Multimodal Data Augmentation [93.05614922383822]
  Attribute-based Multimodal Data Augmentation (ARMADA) is a novel multimodal data augmentation method via knowledge-guided manipulation of visual attributes. ARMADA is a novel multimodal data generation framework that: (i) extracts knowledge-grounded attributes from symbolic KBs for semantically consistent yet distinctive image-text pair generation. This also highlights the need to leverage external knowledge proxies for enhanced interpretability and real-world grounding.
  arXiv  Detail & Related papers  (2024-08-19T15:27:25Z)
• Multi-modal Learnable Queries for Image Aesthetics Assessment [55.28571422062623]
  We propose MMLQ, which utilizes multi-modal learnable queries to extract aesthetics-related features from multi-modal pre-trained features. MMLQ achieves new state-of-the-art performance on multi-modal IAA, beating previous methods by 7.7% and 8.3% in terms of SRCC and PLCC, respectively.
  arXiv  Detail & Related papers  (2024-05-02T14:31:47Z)
• Spatio-Temporal Side Tuning Pre-trained Foundation Models for Video-based Pedestrian Attribute Recognition [58.79807861739438]
  Existing pedestrian recognition (PAR) algorithms are mainly developed based on a static image. We propose to understand human attributes using video frames that can fully use temporal information.
  arXiv  Detail & Related papers  (2024-04-27T14:43:32Z)
• Multi-modal Attribute Prompting for Vision-Language Models [40.39559705414497]
  Pre-trained Vision-Language Models (VLMs) exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. We propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment.
  arXiv  Detail & Related papers  (2024-03-01T01:28:10Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models [9.261590575715669]
  We develop anAttribute-Guided Prompt Tuning (ArGue) for Vision-Language (V&L) models. We align the model with primitive visual attributes generated by Large Language Models (LLMs) Our method significantly outperforms current state-of-the-art prompt tuning methods on both novel class prediction and out-of-distribution generalization tasks.
  arXiv  Detail & Related papers  (2023-11-27T10:34:44Z)
• Attribute-Aware Deep Hashing with Self-Consistency for Large-Scale Fine-Grained Image Retrieval [65.43522019468976]
  We propose attribute-aware hashing networks with self-consistency for generating attribute-aware hash codes. We develop an encoder-decoder structure network of a reconstruction task to unsupervisedly distill high-level attribute-specific vectors. Our models are equipped with a feature decorrelation constraint upon these attribute vectors to strengthen their representative abilities.
  arXiv  Detail & Related papers  (2023-11-21T08:20:38Z)
• Attribute-Guided Multi-Level Attention Network for Fine-Grained Fashion Retrieval [27.751399400911932]
  We introduce an attribute-guided multi-level attention network (AG-MAN) for fine-grained fashion retrieval. Specifically, we first enhance the pre-trained feature extractor to capture multi-level image embedding. Then, we propose a classification scheme where images with the same attribute, albeit with different values, are categorized into the same class.
  arXiv  Detail & Related papers  (2022-12-27T05:28:38Z)
• Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards [41.04264950157818]
  The goal of this work is to develop a novel learning framework for accurate and expressive fashion captioning. We seed the description of an item by first identifying its attributes, and introduce attribute-level semantic (ALS) reward and sentence-level semantic (SLS) reward as metrics to improve the quality of text descriptions.
  arXiv  Detail & Related papers  (2020-08-06T14:52:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.