PRISM: Product Retrieval In Shopping Carts using Hybrid Matching

• URL: http://arxiv.org/abs/2509.14985v1
• Date: Thu, 18 Sep 2025 14:15:37 GMT
• Title: PRISM: Product Retrieval In Shopping Carts using Hybrid Matching
• Abstract summary: We propose a new, hybrid method, called PRISM, for product retrieval in retail settings.<n>PRISM uses vision-language model-based and pixel-wise matching approaches.<n>Our proposed PRISM outperforms the state-of-the-art image retrieval methods by 4.21% in top-1 accuracy.
• Score: 17.39545494939158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compared to traditional image retrieval tasks, product retrieval in retail settings is even more challenging. Products of the same type from different brands may have highly similar visual appearances, and the query image may be taken from an angle that differs significantly from view angles of the stored catalog images. Foundational models, such as CLIP and SigLIP, often struggle to distinguish these subtle but important local differences. Pixel-wise matching methods, on the other hand, are computationally expensive and incur prohibitively high matching times. In this paper, we propose a new, hybrid method, called PRISM, for product retrieval in retail settings by leveraging the advantages of both vision-language model-based and pixel-wise matching approaches. To provide both efficiency/speed and finegrained retrieval accuracy, PRISM consists of three stages: 1) A vision-language model (SigLIP) is employed first to retrieve the top 35 most semantically similar products from a fixed gallery, thereby narrowing the search space significantly; 2) a segmentation model (YOLO-E) is applied to eliminate background clutter; 3) fine-grained pixel-level matching is performed using LightGlue across the filtered candidates. This framework enables more accurate discrimination between products with high inter-class similarity by focusing on subtle visual cues often missed by global models. Experiments performed on the ABV dataset show that our proposed PRISM outperforms the state-of-the-art image retrieval methods by 4.21% in top-1 accuracy while still remaining within the bounds of real-time processing for practical retail deployments.

Related papers

• Fine-Grained Food Image Understanding via Target-Aware Data Alignment [7.633051255808158]
  We propose a data-centric multimodal alignment method for fine-grained food description and recognition.<n>Our method first performs target-aware data selection to identify visually relevant training subsets, then applies VLM-based caption refinement to generate visually grounded, target-style descriptions.
  arXiv  Detail & Related papers  (2026-07-28T14:44:56Z)
• DetailAnywhere: Fashion Detail Generation via Cross-Modal Feature Alignment Distillation [13.86318416278238]
  Diffusion-based generative AI has achieved remarkable success in e-commerce applications such as virtual try-on, poster generation, and product background synthesis.<n>We formalize a new non-template task: Fashion Detail Generation with focus conditioning, and release FDBench.<n> FDBench is the first benchmark comprising 40K+ human-verified reference-detail pairs across 41 different categories.
  arXiv  Detail & Related papers  (2026-07-02T14:26:47Z)
• S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models [8.92478226038086]
  We introduce a learning framework that constructs multi-image preference data across three hierarchical reasoning levels.<n>Our approach maintains strong single-image reasoning performance while simultaneously strengthening multi-image understanding capabilities.
  arXiv  Detail & Related papers  (2026-04-20T17:06:20Z)
• Leveraging Hierarchical Image-Text Misalignment for Universal Fake Image Detection [58.927873049646024]
  We show that fake images cannot be properly aligned with corresponding captions compared to real images.<n>We propose a simple yet effective ITEM by leveraging the image-text misalignment in a joint visual-language space as discriminative clues.
  arXiv  Detail & Related papers  (2025-11-01T06:51:14Z)
• Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions [81.33113485830711]
  We introduce a vision-free, single-encoder retrieval pipeline for vision-language models.<n>We migrate to a text-to-text paradigm with the assistance of VLLM-generated structured image descriptions.<n>Our approach achieves state-of-the-art zero-shot performance on multiple retrieval and compositionality benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T16:22:27Z)
• Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learning [69.33115351856785]
  We present a novel method, called T2I-PAL, to tackle the modality gap issue when using only text captions for PEFT.<n>The core design of T2I-PAL is to leverage pre-trained text-to-image generation models to generate photo-realistic and diverse images from text captions.<n>Extensive experiments on multiple benchmarks, including MS-COCO, VOC2007, and NUS-WIDE, show that our T2I-PAL can boost recognition performance by 3.47% in average.
  arXiv  Detail & Related papers  (2025-06-12T11:09:49Z)
• Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignment [0.7499722271664144]
  Contrastive Language and Image Pairing (CLIP) is a transformative method in multimedia retrieval. CLIP typically trains two neural networks concurrently to generate joint embeddings for text and image pairs. This paper addresses the challenge of optimizing CLIP models for various image-based similarity search scenarios.
  arXiv  Detail & Related papers  (2024-09-03T14:33:01Z)
• Matryoshka Multimodal Models [92.41824727506751]
  We propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens. We find that COCO-style benchmarks only need around 9 visual tokens to obtain accuracy similar to that of using all 576 tokens.
  arXiv  Detail & Related papers  (2024-05-27T17:59:56Z)
• Automated Virtual Product Placement and Assessment in Images using Diffusion Models [1.63075356372232]
  This paper introduces a novel three-stage fully automated VPP system. In the first stage, a language-guided image segmentation model identifies optimal regions within images for product inpainting. In the second stage, Stable Diffusion (SD), fine-tuned with a few example product images, is used to inpaint the product into the previously identified candidate regions. The final stage introduces an "Alignment Module", which is designed to effectively sieve out low-quality images.
  arXiv  Detail & Related papers  (2024-05-02T09:44:13Z)
• Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation [36.992698016947486]
  Open-vocabulary semantic segmentation (OVS) aims to segment images of arbitrary categories specified by class labels or captions. Previous best-performing methods suffer from false matches between image features and category labels. We propose a novel relation-aware intra-modal matching framework for OVS based on visual foundation models.
  arXiv  Detail & Related papers  (2024-03-30T06:29:59Z)
• Pairwise Similarity Learning is SimPLE [104.14303849615496]
  We focus on a general yet important learning problem, pairwise similarity learning (PSL) PSL subsumes a wide range of important applications, such as open-set face recognition, speaker verification, image retrieval and person re-identification. We propose a surprisingly simple proxy-free method, called SimPLE, which requires neither feature/proxy normalization nor angular margin.
  arXiv  Detail & Related papers  (2023-10-13T23:56:47Z)
• Visually Similar Products Retrieval for Shopsy [0.0]
  We design a visual search system for reseller commerce using a multi-task learning approach. Our model consists of three different tasks: attribute classification, triplet ranking and variational autoencoder (VAE)
  arXiv  Detail & Related papers  (2022-10-10T10:59:18Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Devil's in the Details: Aligning Visual Clues for Conditional Embedding in Person Re-Identification [94.77172127405846]
  We propose two key recognition patterns to better utilize the detail information of pedestrian images. CACE-Net achieves state-of-the-art performance on three public datasets.
  arXiv  Detail & Related papers  (2020-09-11T06:28:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.