Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval

• URL: http://arxiv.org/abs/2407.16248v3
• Date: Mon, 5 Aug 2024 09:05:59 GMT
• Title: Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval
• Authors: Xiaowan Hu, Yiyi Chen, Yan Li, Minquan Wang, Haoqian Wang, Quan Chen, Han Li, Peng Jiang,
• Abstract summary: We propose a text-guided attention mechanism that leverages the spoken content of salespeople to guide the model to focus toward intended products. A long-rangetemporal graph network is further designed to achieve both instance-level interaction and frame-level matching. We demonstrate the superior performance of our proposed SGMN model, surpassing the state-of-the-art methods by a substantial margin.
• Score: 32.478352606125306
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid expansion of e-commerce, more consumers have become accustomed to making purchases via livestreaming. Accurately identifying the products being sold by salespeople, i.e., livestreaming product retrieval (LPR), poses a fundamental and daunting challenge. The LPR task encompasses three primary dilemmas in real-world scenarios: 1) the recognition of intended products from distractor products present in the background; 2) the video-image heterogeneity that the appearance of products showcased in live streams often deviates substantially from standardized product images in stores; 3) there are numerous confusing products with subtle visual nuances in the shop. To tackle these challenges, we propose the Spatiotemporal Graphing Multi-modal Network (SGMN). First, we employ a text-guided attention mechanism that leverages the spoken content of salespeople to guide the model to focus toward intended products, emphasizing their salience over cluttered background products. Second, a long-range spatiotemporal graph network is further designed to achieve both instance-level interaction and frame-level matching, solving the misalignment caused by video-image heterogeneity. Third, we propose a multi-modal hard example mining, assisting the model in distinguishing highly similar products with fine-grained features across the video-image-text domain. Through extensive quantitative and qualitative experiments, we demonstrate the superior performance of our proposed SGMN model, surpassing the state-of-the-art methods by a substantial margin. The code is available at https://github.com/Huxiaowan/SGMN.

Related papers

• Vivid-ZOO: Multi-View Video Generation with Diffusion Model [76.96449336578286]
  New challenges lie in the lack of massive captioned multi-view videos and the complexity of modeling such multi-dimensional distribution. We propose a novel diffusion-based pipeline that generates high-quality multi-view videos centered around a dynamic 3D object from text.
  arXiv  Detail & Related papers  (2024-06-12T21:44:04Z)
• Matryoshka Multimodal Models [92.41824727506751]
  We propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens. We find that COCO-style benchmarks only need around 9 visual tokens to obtain accuracy similar to that of using all 576 tokens.
  arXiv  Detail & Related papers  (2024-05-27T17:59:56Z)
• MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization [93.5217515566437]
  Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics. Existing MPS methods can produce promising results, but they still lack end-to-end product summarization. We propose an end-to-end multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce.
  arXiv  Detail & Related papers  (2023-08-22T11:00:09Z)
• Cross-Domain Product Representation Learning for Rich-Content E-Commerce [16.418118040661646]
  This paper introduces a large-scale cRoss-dOmain Product Ecognition dataset, called ROPE. ROPE covers a wide range of product categories and contains over 180,000 products, corresponding to millions of short videos and live streams. It is the first dataset to cover product pages, short videos, and live streams simultaneously, providing the basis for establishing a unified product representation across different media domains.
  arXiv  Detail & Related papers  (2023-08-10T13:06:05Z)
• Cross-view Semantic Alignment for Livestreaming Product Recognition [24.38606354376169]
  We present LPR4M, a large-scale multimodal dataset that covers 34 categories. LPR4M contains diverse videos and noise modality pairs while exhibiting a long-tailed distribution. A novel Patch Feature Reconstruction loss is proposed to penalize the semantic misalignment between cross-view patches.
  arXiv  Detail & Related papers  (2023-08-09T12:23:41Z)
• Multi-queue Momentum Contrast for Microvideo-Product Retrieval [57.527227171945796]
  We formulate the microvideo-product retrieval task, which is the first attempt to explore the retrieval between the multi-modal and multi-modal instances. A novel approach named Multi-Queue Momentum Contrast (MQMC) network is proposed for bidirectional retrieval. A discriminative selection strategy with a multi-queue is used to distinguish the importance of different negatives based on their categories.
  arXiv  Detail & Related papers  (2022-12-22T03:47:14Z)
• Visually Similar Products Retrieval for Shopsy [0.0]
  We design a visual search system for reseller commerce using a multi-task learning approach. Our model consists of three different tasks: attribute classification, triplet ranking and variational autoencoder (VAE)
  arXiv  Detail & Related papers  (2022-10-10T10:59:18Z)
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining [108.86502855439774]
  We investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval. We contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. We propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE)
  arXiv  Detail & Related papers  (2021-07-30T12:11:24Z)
• Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce [18.651201334846352]
  We present an innovative demonstration of multi-modal retrieval system called "Fashion Focus" It enables to exactly localize the product images in the online video as the focuses. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching.
  arXiv  Detail & Related papers  (2021-02-09T09:45:04Z)
• Comprehensive Information Integration Modeling Framework for Video Titling [124.11296128308396]
  We integrate comprehensive sources of information, including the content of consumer-generated videos, the narrative comment sentences supplied by consumers, and the product attributes, in an end-to-end modeling framework. To tackle this issue, the proposed method consists of two processes, i.e., granular-level interaction modeling and abstraction-level story-line summarization. We collect a large-scale dataset accordingly from real-world data in Taobao, a world-leading e-commerce platform.
  arXiv  Detail & Related papers  (2020-06-24T10:38:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.