Native Multimodal Representation Learning for Click-Through Rate Prediction in E-Commerce Scenarios

• URL: http://arxiv.org/abs/2608.24091v1
• Date: Tue, 25 Aug 2026 05:36:17 GMT
• Title: Native Multimodal Representation Learning for Click-Through Rate Prediction in E-Commerce Scenarios
• Abstract summary: Multimodal representations have been widely adopted in industrial e-commerce recommendation systems.<n>We propose a Mine-Then-Train method that mines high-quality, multimodally interpretable training samples from CTR data.
• Score: 12.746111237120289
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal representations have been widely adopted in industrial e-commerce recommendation systems. Due to their strong semantic understanding and generalization capabilities, they enhance the performance of traditional sparse ID-based Click-Through Rate (CTR) prediction models. Current multimodal application frameworks in the CTR prediction task typically follow a two-stage paradigm: first, pre-training a multimodal encoder on data from specific recommendation scenarios; second, extracting items' multimodal representations using this pre-trained multimodal encoder and integrating them into the CTR prediction model. However, the training objectives and data distribution of multimodal pre-training tasks often differ from those of the CTR prediction task, which limits the effectiveness of multimodal representation on downstream tasks. In this paper, we focus on how to learn Native Multimodal Representation for the CTR prediction task. One intuitive solution is to jointly train the multimodal encoder and CTR model end-to-end on the CTR task, with the expectation that the encoder can automatically learn downstream-relevant knowledge. However, we find that the end-to-end training does not bring performance improvements to existing multimodal application paradigms. Our analysis reveals that user behaviors in raw CTR data are driven by both multimodal semantics and non-multimodal factors, leading to ambiguous supervision and inconsistent encoder updates. To address this, we propose a Mine-Then-Train method that mines high-quality, multimodally interpretable training samples from CTR data and uses them to fine-tune the multimodal encoder for better alignment with user click preferences. Offline and online experiments demonstrate the effectiveness of our approach.

Related papers

• DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning [64.33887406863899]
  Multimodal deception detection aims to identify deceptive behavior by analyzing audiovisual cues for forensics and security.<n>Existing benchmarks provide only binary labels without intermediate reasoning cues.<n>We construct reasoning datasets with structured cue-level descriptions and reasoning chains.<n>With 1695 samples, it is the largest non-laboratory deception detection dataset.
  arXiv  Detail & Related papers  (2026-03-25T04:06:36Z)
• LADLE-MM: Limited Annotation based Detector with Learned Ensembles for Multimodal Misinformation [8.769506450302154]
  LADLE-MM is a model-soup multimodal misinformation detector with Learned Ensembles for Multimodal Misinformation.<n>It is composed of two unimodal branches and a third multimodal one that enhances image and text representations.<n>It achieves competitive performance on both binary and multi-label classification tasks.
  arXiv  Detail & Related papers  (2025-12-23T11:14:58Z)
• MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks [11.359741665798195]
  This paper explores a more flexible multi-modal, multi-task pre-training strategy for Earth Observation (EO) data.<n>Specifically, we adopt a Multi-modal Multi-task Masked Autoencoder (MultiMAE) that we pre-train by reconstructing diverse input modalities.<n>Our approach exhibits significant flexibility, handling diverse input configurations without requiring modality-specific pre-trained models.
  arXiv  Detail & Related papers  (2025-05-20T22:24:36Z)
• Feature Fusion Revisited: Multimodal CTR Prediction for MMCTR Challenge [4.3058911704400415]
  The EReL@MIR workshop provided a valuable opportunity to experiment with various approaches aimed at improving the efficiency of multimodal representation learning.<n>Our team was honored to receive the award for Task 2 - Winner (Multimodal CTR Prediction)
  arXiv  Detail & Related papers  (2025-04-26T16:04:33Z)
• Quadratic Interest Network for Multimodal Click-Through Rate Prediction [12.989347150912685]
  Multimodal click-through rate (CTR) prediction is a key technique in industrial recommender systems.<n>We propose a novel model for Task 2, named Quadratic Interest Network (QIN) for Multimodal CTR Prediction.
  arXiv  Detail & Related papers  (2025-04-24T16:08:52Z)
• MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks [50.98856172702256]
  We propose the Modality-INformed knowledge Distillation (MIND) framework, a multimodal model compression approach.<n>MIND transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student.<n>We evaluate MIND on binary and multilabel clinical prediction tasks using time series data and chest X-ray images.
  arXiv  Detail & Related papers  (2025-02-03T08:50:00Z)
• MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying [110.83590008788745]
  Motion prediction is crucial for autonomous driving systems to understand complex driving scenarios and make informed decisions. In this paper, we propose Motion TRansformer (MTR) frameworks to address these challenges. The initial MTR framework utilizes a transformer encoder-decoder structure with learnable intention queries. We introduce an advanced MTR++ framework, extending the capability of MTR to simultaneously predict multimodal motion for multiple agents.
  arXiv  Detail & Related papers  (2023-06-30T16:23:04Z)
• Routing with Self-Attention for Multimodal Capsule Networks [108.85007719132618]
  We present a new multimodal capsule network that allows us to leverage the strength of capsules in the context of a multimodal learning framework. To adapt the capsules to large-scale input data, we propose a novel routing by self-attention mechanism that selects relevant capsules. This allows not only for robust training with noisy video data, but also to scale up the size of the capsule network compared to traditional routing methods.
  arXiv  Detail & Related papers  (2021-12-01T19:01:26Z)
• Unsupervised Multimodal Language Representations using Convolutional Autoencoders [5.464072883537924]
  We propose extracting unsupervised Multimodal Language representations that are universal and can be applied to different tasks. We map the word-level aligned multimodal sequences to 2-D matrices and then use Convolutional Autoencoders to learn embeddings by combining multiple datasets. It is also shown that our method is extremely lightweight and can be easily generalized to other tasks and unseen data with small performance drop and almost the same number of parameters.
  arXiv  Detail & Related papers  (2021-10-06T18:28:07Z)
• Joint predictions of multi-modal ride-hailing demands: a deep multi-task multigraph learning-based approach [64.18639899347822]
  We propose a deep multi-task multi-graph learning approach, which combines multiple multi-graph convolutional (MGC) networks for predicting demands for different service modes. We show that our propose approach outperforms the benchmark algorithms in prediction accuracy for different ride-hailing modes.
  arXiv  Detail & Related papers  (2020-11-11T07:10:50Z)
• SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction [72.37440317774556]
  We propose advances that address two key challenges in future trajectory prediction. multimodality in both training data and predictions and constant time inference regardless of number of agents.
  arXiv  Detail & Related papers  (2020-07-26T08:17:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.