AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce

• URL: http://arxiv.org/abs/2604.20135v1
• Date: Wed, 22 Apr 2026 03:02:21 GMT
• Title: AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce
• Abstract summary: We propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL)<n>It defines product fine-grained understanding as an attribute generation task.<n>It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text.
• Score: 16.733243065736023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal representation is crucial for E-commerce tasks such as identical product retrieval. Large representation models (e.g., VLM2Vec) demonstrate strong multimodal understanding capabilities, yet they struggle with fine-grained semantic comprehension, which is essential for distinguishing highly similar items. To address this, we propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL), which defines product fine-grained understanding as an attribute generation task. It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text, and enhances representation learning through a two-stage training framework: 1) Attribute-Guided Contrastive Learning (AGCL), where the key attributes generated by the MLLM are used in the image-text contrastive learning training process to identify hard samples and filter out noisy false negatives. 2) Retrieval-aware Attribute Reinforcement (RAR), where the improved retrieval performance of the representation model post-attribute integration serves as a reward signal to enhance MLLM's attribute generation during multimodal fine-tuning. Extensive experiments on large-scale E-commerce datasets demonstrate that our method achieves state-of-the-art performance on multiple downstream retrieval tasks, validating the effectiveness of harnessing generative models to advance fine-grained representation learning.

Related papers

• MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search [9.467425331289988]
  Multiplex Multimodal Representation Model (MMRM) is a unified framework that aligns Multimodal Large Language Models with diverse collaborative signals.<n>MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.
  arXiv  Detail & Related papers  (2026-07-13T02:54:04Z)
• Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs [37.35588113092876]
  We propose FSAR-LLaVA, the first end-to-end method to leverage MLLMs as a multimodal knowledge base for enhancing FSAR.<n>First, we leverage the MLLM's multimodal decoder to extracttemporally and semantically enriched representations.<n>Next, we leverage the versatility of MLLMs to craft input prompts that flexibly adapt to diverse scenarios.<n>Finally, to enable multimodal features to guide metric learning jointly, we introduce a training-free Multimodal Prototype Matching Metric.
  arXiv  Detail & Related papers  (2026-03-27T03:05:56Z)
• MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding [19.89836326556511]
  We argue that generative Multimodal Large Language Models hold significant potential for improving product representation learning.<n>We propose the first generative MLLM-based model named MOON for product representation learning.<n>Our method employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content.
  arXiv  Detail & Related papers  (2025-08-16T09:59:25Z)
• Multi-modal Relational Item Representation Learning for Inferring Substitutable and Complementary Items [10.98931494075836]
  We introduce a novel self-supervised multi-modal relational item representation learning framework designed to infer substitutable and complementary items.<n>MMSC consists of three main components: (1) a multi-modal item representation learning module that leverages a multi-modal foundational model and learns from item metadata, (2) a self-supervised behavior-based representation learning module that denoises and learns from user behavior data, and (3) a hierarchical representation aggregation mechanism that integrates item representations at both the semantic and task levels.
  arXiv  Detail & Related papers  (2025-07-29T22:38:39Z)
• Learning Item Representations Directly from Multimodal Features for Effective Recommendation [51.49251689107541]
  multimodal recommender systems predominantly leverage Bayesian Personalized Ranking (BPR) optimization to learn item representations.<n>We propose a novel model (i.e., LIRDRec) that learns item representations directly from multimodal features to augment recommendation performance.
  arXiv  Detail & Related papers  (2025-05-08T05:42:22Z)
• EMMA: Efficient Visual Alignment in Multi-Modal LLMs [56.03417732498859]
  EMMA is a lightweight cross-modality module designed to efficiently fuse visual and textual encodings.<n>EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.
  arXiv  Detail & Related papers  (2024-10-02T23:00:31Z)
• Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models [49.439311430360284]
  We introduce a novel data synthesis method inspired by contrastive learning and image difference captioning.<n>Our key idea involves challenging the model to discern both matching and distinct elements.<n>We leverage this generated dataset to fine-tune state-of-the-art (SOTA) MLLMs.
  arXiv  Detail & Related papers  (2024-08-08T17:10:16Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization [93.5217515566437]
  Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics. Existing MPS methods can produce promising results, but they still lack end-to-end product summarization. We propose an end-to-end multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce.
  arXiv  Detail & Related papers  (2023-08-22T11:00:09Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce [9.46186546774799]
  We propose a contrastive learning framework that aligns language and visual models using unlabeled raw product text and images. We present techniques we used to train large-scale representation learning models and share solutions that address domain-specific challenges.
  arXiv  Detail & Related papers  (2022-07-01T05:16:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.