MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding

• URL: http://arxiv.org/abs/2508.11999v1
• Date: Sat, 16 Aug 2025 09:59:25 GMT
• Title: MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding
• Authors: Daoze Zhang, Zhanheng Nie, Jianyu Liu, Chenghan Fu, Wanxian Guan, Yuan Gao, Jun Song, Pengjie Wang, Jian Xu, Bo Zheng,
• Abstract summary: We argue that generative Multimodal Large Language Models hold significant potential for improving product representation learning.<n>We propose the first generative MLLM-based model named MOON for product representation learning.<n>Our method employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content.
• Score: 19.89836326556511
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancement of e-commerce, exploring general representations rather than task-specific ones has attracted increasing research attention. For product understanding, although existing discriminative dual-flow architectures drive progress in this field, they inherently struggle to model the many-to-one alignment between multiple images and texts of products. Therefore, we argue that generative Multimodal Large Language Models (MLLMs) hold significant potential for improving product representation learning. Nevertheless, achieving this goal still remains non-trivial due to several key challenges: the lack of multimodal and aspect-aware modeling modules in typical LLMs; the common presence of background noise in product images; and the absence of a standard benchmark for evaluation. To address these issues, we propose the first generative MLLM-based model named MOON for product representation learning. Our method (1) employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content; (2) effectively detects core semantic regions in product images to mitigate the distraction and interference caused by background noise; and (3) introduces the specialized negative sampling strategy to increase the difficulty and diversity of negative samples. In addition, we release a large-scale multimodal benchmark MBE for various product understanding tasks. Experimentally, our model demonstrates competitive zero-shot performance on both our benchmark and the public dataset, showcasing strong generalization across various downstream tasks, including cross-modal retrieval, product classification, and attribute prediction. Furthermore, the case study and visualization illustrate the effectiveness of MOON for product understanding.

Related papers

• Benchmarking Multimodal Large Language Models for Missing Modality Completion in Product Catalogues [19.732113077201326]
  Missing-modality information on e-commerce platforms, such as absent product images or textual descriptions, often arises from annotation errors or incomplete metadata.<n>This work investigates the question: can Multimodal Large Language Models generate missing modalities for products in e-commerce scenarios?<n>We propose the Missing Modality Product Completion Benchmark (MMPCBench), which consists of two sub-benchmarks: a Content Quality Completion Benchmark and a Recommendation Benchmark.<n>We evaluate six state-of-the-art MLLMs from the Qwen2.5-VL and Gemma-3 model families across nine real-world e-commerce categories, focusing
  arXiv  Detail & Related papers  (2026-01-27T16:13:26Z)
• Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space [52.34072027212278]
  Embedding models are a fundamental component of modern AI systems such as semantic search and retrieval-augmented generation.<n>Recent advances in large foundation models have substantially accelerated the development of embedding models.<n>We present the first systematic study of converting Multimodal dLLMs into embedding models.
  arXiv  Detail & Related papers  (2026-01-19T06:51:15Z)
• MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding [11.989986738179427]
  MOON2.0 is a dynamic modality-balanced representation learning framework for e-commerce product understanding.<n>MoE module adaptively processes input samples by their modality composition, enabling Multimodal Joint Learning.<n> MBE2.0 is a co-augmented multimodal representation benchmark for e-commerce representation learning and evaluation.
  arXiv  Detail & Related papers  (2025-11-16T04:29:35Z)
• Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images [58.553448128258566]
  This paper bridges the dual gaps in large-scale high-quality data and capability enhancement methodologies.<n>We introduce STAR-64K, a dataset comprising 64K high-quality multi-modal instruction samples, and conduct experiments across 5 open-source MLLMs.
  arXiv  Detail & Related papers  (2025-10-22T02:23:40Z)
• Multi-modal Relational Item Representation Learning for Inferring Substitutable and Complementary Items [10.98931494075836]
  We introduce a novel self-supervised multi-modal relational item representation learning framework designed to infer substitutable and complementary items.<n>MMSC consists of three main components: (1) a multi-modal item representation learning module that leverages a multi-modal foundational model and learns from item metadata, (2) a self-supervised behavior-based representation learning module that denoises and learns from user behavior data, and (3) a hierarchical representation aggregation mechanism that integrates item representations at both the semantic and task levels.
  arXiv  Detail & Related papers  (2025-07-29T22:38:39Z)
• Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging [103.98582374569789]
  Model merging aims to combine multiple expert models into a single model, thereby reducing storage and serving costs.<n>Previous studies have primarily focused on merging visual classification models or Large Language Models (LLMs) for code and math tasks.<n>We introduce the model merging benchmark for MLLMs, which includes multiple tasks such as VQA, Geometry, Chart, OCR, and Grounding, providing both LoRA and full fine-tuning models.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models [23.916205754112774]
  Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks.<n>We propose TAMP, a simple yet effective pruning framework tailored for MLLMs.<n>We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities.
  arXiv  Detail & Related papers  (2025-04-14T05:44:38Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• Model Composition for Multimodal Large Language Models [71.5729418523411]
  We propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters.
  arXiv  Detail & Related papers  (2024-02-20T06:38:10Z)
• MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization [93.5217515566437]
  Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics. Existing MPS methods can produce promising results, but they still lack end-to-end product summarization. We propose an end-to-end multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce.
  arXiv  Detail & Related papers  (2023-08-22T11:00:09Z)
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining [108.86502855439774]
  We investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval. We contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. We propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE)
  arXiv  Detail & Related papers  (2021-07-30T12:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.