MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding

• URL: http://arxiv.org/abs/2406.10701v1
• Date: Sat, 15 Jun 2024 17:56:09 GMT
• Title: MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding
• Authors: Baixuan Xu, Weiqi Wang, Haochen Shi, Wenxuan Ding, Huihao Jing, Tianqing Fang, Jiaxin Bai, Long Chen, Yangqiu Song,
• Abstract summary: MIND is a framework that infers purchase intentions from multimodal product metadata and prioritizes human-centric ones. Using Amazon Review data, we create a multimodal intention knowledge base, which contains 1,264,441 million intentions. Our obtained intentions significantly enhance large language models in two intention comprehension tasks.
• Score: 45.47495643376656
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Improving user experience and providing personalized search results in E-commerce platforms heavily rely on understanding purchase intention. However, existing methods for acquiring large-scale intentions bank on distilling large language models with human annotation for verification. Such an approach tends to generate product-centric intentions, overlook valuable visual information from product images, and incurs high costs for scalability. To address these issues, we introduce MIND, a multimodal framework that allows Large Vision-Language Models (LVLMs) to infer purchase intentions from multimodal product metadata and prioritize human-centric ones. Using Amazon Review data, we apply MIND and create a multimodal intention knowledge base, which contains 1,264,441 million intentions derived from 126,142 co-buy shopping records across 107,215 products. Extensive human evaluations demonstrate the high plausibility and typicality of our obtained intentions and validate the effectiveness of our distillation framework and filtering mechanism. Additional experiments reveal that our obtained intentions significantly enhance large language models in two intention comprehension tasks.

Related papers

• IntentionQA: A Benchmark for Evaluating Purchase Intention Comprehension Abilities of Language Models in E-commerce [50.41970803871156]
  In this paper, we present IntentionQA, a benchmark to evaluate LMs' comprehension of purchase intentions in E-commerce. IntentionQA consists of 4,360 carefully curated problems across three difficulty levels, constructed using an automated pipeline. Human evaluations demonstrate the high quality and low false-negative rate of our benchmark.
  arXiv  Detail & Related papers  (2024-06-14T16:51:21Z)
• Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model [82.93634081255942]
  Vision-language connector plays a crucial role to link the pre-trained vision encoders with Large Language Models (LLMs) We propose a strong vision-language connector that enables MLLMs to achieve high accuracy while maintain low cost. We introduce the Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining.
  arXiv  Detail & Related papers  (2024-05-28T04:23:00Z)
• Multimodal Large Language Models to Support Real-World Fact-Checking [80.41047725487645]
  Multimodal large language models (MLLMs) carry the potential to support humans in processing vast amounts of information. While MLLMs are already being used as a fact-checking tool, their abilities and limitations in this regard are understudied. We propose a framework for systematically assessing the capacity of current multimodal models to facilitate real-world fact-checking.
  arXiv  Detail & Related papers  (2024-03-06T11:32:41Z)
• Multi-Objective Personalized Product Retrieval in Taobao Search [27.994166796745496]
  We propose a novel Multi-Objective Personalized Product Retrieval (MOPPR) model with four hierarchical optimization objectives: relevance, exposure, click and purchase. MOPPR achieves 0.96% transaction and 1.29% GMV improvements in a 28-day online A/B test. Since the Double-11 shopping festival of 2021, MOPPR has been fully deployed in mobile Taobao search, replacing the previous MGDSPR.
  arXiv  Detail & Related papers  (2022-10-09T05:18:42Z)
• e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce [9.46186546774799]
  We propose a contrastive learning framework that aligns language and visual models using unlabeled raw product text and images. We present techniques we used to train large-scale representation learning models and share solutions that address domain-specific challenges.
  arXiv  Detail & Related papers  (2022-07-01T05:16:47Z)
• ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest [60.841761065439414]
  At Pinterest, we build a single set of product embeddings called ItemSage to provide relevant recommendations in all shopping use cases. This approach has led to significant improvements in engagement and conversion metrics, while reducing both infrastructure and maintenance cost.
  arXiv  Detail & Related papers  (2022-05-24T02:28:58Z)
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining [108.86502855439774]
  We investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval. We contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. We propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE)
  arXiv  Detail & Related papers  (2021-07-30T12:11:24Z)
• A Multimodal Late Fusion Model for E-Commerce Product Classification [7.463657960984954]
  In this study, we investigated a multimodal late fusion approach based on text and image modalities to categorize e-commerce products on Rakuten. Specifically, we developed modal specific state-of-the-art deep neural networks for each input modal, and then fused them at the decision level. Our team named pa_curis won the 1st place with a macro-F1 of 0.9144 on the final leaderboard.
  arXiv  Detail & Related papers  (2020-08-14T03:46:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.