Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace

• URL: http://arxiv.org/abs/2508.05661v1
• Date: Thu, 31 Jul 2025 05:13:20 GMT
• Title: Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace
• Authors: Andre Rusli, Shoma Ishimoto, Sho Akiyama, Aman Kumar Singh,
• Abstract summary: This paper presents a scalable visual search system deployed in Mercari's C2C marketplace.<n>We evaluate recent vision-language models for zero-shot image retrieval and compare their performance with an existing fine-tuned baseline.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Visual search offers an intuitive way for customers to explore diverse product catalogs, particularly in consumer-to-consumer (C2C) marketplaces where listings are often unstructured and visually driven. This paper presents a scalable visual search system deployed in Mercari's C2C marketplace, where end-users act as buyers and sellers. We evaluate recent vision-language models for zero-shot image retrieval and compare their performance with an existing fine-tuned baseline. The system integrates real-time inference and background indexing workflows, supported by a unified embedding pipeline optimized through dimensionality reduction. Offline evaluation using user interaction logs shows that the multilingual SigLIP model outperforms other models across multiple retrieval metrics, achieving a 13.3% increase in nDCG@5 over the baseline. A one-week online A/B test in production further confirms real-world impact, with the treatment group showing substantial gains in engagement and conversion, up to a 40.9% increase in transaction rate via image search. Our findings highlight that recent zero-shot models can serve as a strong and practical baseline for production use, which enables teams to deploy effective visual search systems with minimal overhead, while retaining the flexibility to fine-tune based on future data or domain-specific needs.

Related papers

• From Pixels to Purchase: Building and Evaluating a Taxonomy-Decoupled Visual Search Engine for Home Goods E-commerce [6.200631104634354]
  Visual search is critical for e-commerce, especially in style-driven domains where user intent is subjective and open-ended.<n>We propose a taxonomy-decoupled architecture that uses classification-free region proposals and unified embeddings for similarity retrieval.<n>Our system improves retrieval quality and yields a measurable uplift in customer engagement, while our offline evaluation metrics strongly correlate with real-world outcomes.
  arXiv  Detail & Related papers  (2026-01-16T20:54:30Z)
• OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Search [43.94443394870866]
  OneSearch is the first industrial-deployed end-to-end generative framework for e-commerce search.<n>OneSearch reduces operational expenditure by 75.40% and improves Model FLOPs Utilization from 3.26% to 27.32%.<n>The system has been successfully deployed across multiple search scenarios in Kuaishou, serving millions of users.
  arXiv  Detail & Related papers  (2025-09-03T11:50:04Z)
• Leveraging Generative Models for Real-Time Query-Driven Text Summarization in Large-Scale Web Search [54.987957691350665]
  Query-Driven Text Summarization (QDTS) aims to generate concise and informative summaries from textual documents based on a given query.<n>Traditional extractive summarization models, based primarily on ranking candidate summary segments, have been the dominant approach in industrial applications.<n>We propose a novel framework to pioneer the application of generative models to address real-time QDTS in industrial web search.
  arXiv  Detail & Related papers  (2025-08-28T08:51:51Z)
• Semantic Ads Retrieval at Walmart eCommerce with Language Models Progressively Trained on Multiple Knowledge Domains [6.1008328784394]
  We present an end-to-end solution tailored to optimize the ads retrieval system on Walmart.com.<n>Our approach is to pretrain the BERT-like classification model with product category information.<n>It enhances the search relevance metric by up to 16% compared to a baseline DSSM-based model.
  arXiv  Detail & Related papers  (2025-02-13T09:01:34Z)
• Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models [51.067146460271466]
  Evaluation of visual generative models can be time-consuming and computationally expensive.<n>We propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations.<n>It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools.
  arXiv  Detail & Related papers  (2024-12-10T18:52:39Z)
• Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval [1.6874375111244329]
  State-of-the-art image retrieval systems train specific neural networks for each dataset.<n>Off-the-shelf foundation models fall short in achieving performance comparable to dataset-specific models.<n>We introduce Autoencoders with Strong Variance Constraints (AE-SVC), which significantly improves the performance of foundation models.
  arXiv  Detail & Related papers  (2024-10-09T16:05:16Z)
• Exploring Fine-grained Retail Product Discrimination with Zero-shot Object Classification Using Vision-Language Models [50.370043676415875]
  In smart retail applications, the large number of products and their frequent turnover necessitate reliable zero-shot object classification methods. We introduce the MIMEX dataset, comprising 28 distinct product categories. We benchmark the zero-shot object classification performance of state-of-the-art vision-language models (VLMs) on the proposed MIMEX dataset.
  arXiv  Detail & Related papers  (2024-09-23T12:28:40Z)
• Overview of the TREC 2023 Product Product Search Track [70.56592126043546]
  This is the first year of the TREC Product search track. The focus was the creation of a reusable collection. We leverage the new product search corpus, which includes contextual metadata.
  arXiv  Detail & Related papers  (2023-11-14T02:25:18Z)
• Unified Embedding Based Personalized Retrieval in Etsy Search [0.206242362470764]
  We propose learning a unified embedding model incorporating graph, transformer and term-based embeddings end to end. Our personalized retrieval model significantly improves the overall search experience, as measured by a 5.58% increase in search purchase rate and a 2.63% increase in site-wide conversion rate.
  arXiv  Detail & Related papers  (2023-06-07T23:24:50Z)
• Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval [152.3504607706575]
  This research aims to conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks. We exploit to train a more effective cross-modal model which is adaptively capable of incorporating key concept information from the multi-modal data.
  arXiv  Detail & Related papers  (2022-06-17T15:40:45Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.