MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search

• URL: http://arxiv.org/abs/2607.11030v1
• Date: Mon, 13 Jul 2026 02:54:04 GMT
• Title: MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search
• Abstract summary: Multiplex Multimodal Representation Model (MMRM) is a unified framework that aligns Multimodal Large Language Models with diverse collaborative signals.<n>MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.
• Score: 9.467425331289988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal Large Language Models (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM's superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.

Related papers

• AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce [16.733243065736023]
  We propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL)<n>It defines product fine-grained understanding as an attribute generation task.<n>It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text.
  arXiv  Detail & Related papers  (2026-04-22T03:02:21Z)
• MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model [57.89395815934156]
  Multi-Turn Contrastive Learning (MuCo) is a dialogue-inspired framework that revisits this process.<n>Experiments exhibit MuCo with a newly curated 5M multimodal multi-turn dataset (M3T)
  arXiv  Detail & Related papers  (2026-02-06T05:18:33Z)
• MUSE: A Simple Yet Effective Multimodal Search-Based Framework for Lifelong User Interest Modeling [48.18456242206804]
  We present a systematic analysis of how to leverage multimodal signals across both stages of lifelong modeling framework.<n>We propose MUSE, a simple yet effective multimodal search-based framework.<n>MUSE has been deployed in Taobao display advertising system, enabling 100K-length user behavior sequence modeling.
  arXiv  Detail & Related papers  (2025-12-08T06:55:13Z)
• TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models [23.916205754112774]
  Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks.<n>We propose TAMP, a simple yet effective pruning framework tailored for MLLMs.<n>We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities.
  arXiv  Detail & Related papers  (2025-04-14T05:44:38Z)
• MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks [50.98856172702256]
  We propose the Modality-INformed knowledge Distillation (MIND) framework, a multimodal model compression approach.<n>MIND transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student.<n>We evaluate MIND on binary and multilabel clinical prediction tasks using time series data and chest X-ray images.
  arXiv  Detail & Related papers  (2025-02-03T08:50:00Z)
• A Unified Graph Transformer for Overcoming Isolations in Multi-modal Recommendation [9.720586396359906]
  We argue that existing multi-modal recommender systems typically use isolated processes for both feature extraction and modality modelling. We propose a novel model, called Unified Multi-modal Graph Transformer (UGT), which leverages a multi-way transformer to extract aligned multi-modal features. We show that the UGT model can achieve significant effectiveness gains, especially when jointly optimised with the commonly-used multi-modal recommendation losses.
  arXiv  Detail & Related papers  (2024-07-29T11:04:31Z)
• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
  We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
  arXiv  Detail & Related papers  (2024-05-18T12:16:01Z)
• Model Composition for Multimodal Large Language Models [71.5729418523411]
  We propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters.
  arXiv  Detail & Related papers  (2024-02-20T06:38:10Z)
• MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization [93.5217515566437]
  Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics. Existing MPS methods can produce promising results, but they still lack end-to-end product summarization. We propose an end-to-end multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce.
  arXiv  Detail & Related papers  (2023-08-22T11:00:09Z)
• MM-GEF: Multi-modal representation meet collaborative filtering [43.88159639990081]
  We propose a graph-based item structure enhancement method MM-GEF: Multi-Modal recommendation with Graph Early-Fusion. MM-GEF learns refined item representations by injecting structural information obtained from both multi-modal and collaborative signals.
  arXiv  Detail & Related papers  (2023-08-14T15:47:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.