What Matters for Grocery Product Retrieval with Open Source Vision Language Models

• URL: http://arxiv.org/abs/2605.18029v1
• Date: Mon, 18 May 2026 08:20:13 GMT
• Title: What Matters for Grocery Product Retrieval with Open Source Vision Language Models
• Abstract summary: We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge.<n>Switching from raw web-scrapes to filtered datasets delivers up to 16.6% accuracy gains.<n>State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal product retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that standard vision-language benchmarks fail to capture. We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge, isolating pre-training data, architecture, and input resolution. Our analysis yields three actionable findings. \textbf{(1) Data quality trumps scale.} Switching from raw web-scrapes to filtered datasets delivers up to 16.6\% accuracy gains, exceeding the benefit of doubling model parameters. \textbf{(2) Efficient models can win.} MobileCLIP-B (150M parameters) outperforms 351M counterparts trained on noisy data. We introduce \textit{semantic power density} ($φ$), an efficiency metric that penalizes sub-threshold accuracy. \textbf{(3) A precision gap persists.} State-of-the-art models achieve 94.5\% Recall@5 but suffer a 17.5\% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs. Code and evaluation scripts are available at \url{https://github.com/upeee/openmpr}.

Related papers

• Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation [43.1212452324751]
  Existing datasets in this domain either entirely lack descriptive text, or suffer from severe text-image semantic misalignment.<n>We present a novel methodology to construct and automatically validate a large-scale multimodal dataset for disaster response.
  arXiv  Detail & Related papers  (2026-07-30T14:23:01Z)
• Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models [7.176656402971074]
  We propose a classification-guided large vision-language model (LVLM) framework for multi-type VIE that achieves high accuracy with minimal supervision.<n>The framework offers an efficient, scalable solution for complex document understanding in office automation.
  arXiv  Detail & Related papers  (2026-07-22T03:27:50Z)
• VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection [0.04736448323490553]
  VendorBench-100 is a cross-paradigm benchmark for deepfake image detection.<n>It evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework.<n>Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors.
  arXiv  Detail & Related papers  (2026-07-07T13:22:00Z)
• Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model [47.811446766453834]
  Existing Multi-modal Coreference Resolution (MCR) methods require training with annotated data from the target dataset before they can be applied.<n>We propose a plug-and-adapt method that strategically adapts a carefully pre-trained emphalignment model for immediate use in MCR tasks.
  arXiv  Detail & Related papers  (2026-06-16T14:04:13Z)
• Employing Vision-Language Models for Face Image Quality Assessment [0.023141219541346198]
  Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines.<n>State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes"<n>We investigate the potential of off-the-shelf Vision-Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-05-17T14:57:52Z)
• VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents [1.06378109904813]
  VAREX is a benchmark for evaluating structured data extraction from government forms.<n>The benchmark comprises 1,777 documents with 1,771 unique documents with validated ground truth through three-phase quality assurance.<n>Results reveal that below 4B parameters, compliance output -- not extraction capability -- is a dominant bottleneck.
  arXiv  Detail & Related papers  (2026-03-16T11:15:56Z)
• VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering [53.662676566188175]
  A key bottleneck lies in the lack of public, large-scale, high-quality Scientific Visual Question Answering (SVQA) datasets.<n>We propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context.<n>We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types.
  arXiv  Detail & Related papers  (2025-11-25T04:14:52Z)
• RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models [6.977949425464]
  We leverage an overlooked agglomerative vision foundation model, RADIO, to improve zero-shot OVSS along three key axes simultaneously: mIoU, latency, and parameter efficiency.<n>Our approach, RADSeg, achieves 6-30% mIoU improvement in the base ViT class while being 3.95x faster and using 2.5x fewer parameters.
  arXiv  Detail & Related papers  (2025-11-24T21:15:01Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
  Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
  arXiv  Detail & Related papers  (2025-02-24T07:15:05Z)
• Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classification [6.975902383951604]
  Current methodologies face difficulties with the unpredictable distribution of outliers. We present the Dual for Threshold-Based Re-Classification (DETER) to address these challenges. Our model outperforms previous benchmarks, increasing up to 13% and 5% in F1 score for known and unknown intents.
  arXiv  Detail & Related papers  (2024-05-30T11:46:42Z)
• See, Say, and Segment: Teaching LMMs to Overcome False Premises [67.36381001664635]
  We propose a cascading and joint training approach for LMMs to solve this task. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, and finally "segment" by outputting the mask of the desired objects if they exist.
  arXiv  Detail & Related papers  (2023-12-13T18:58:04Z)
• ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extraction [51.87391234815163]
  E-commerce platforms require structured product data in the form of attribute-value pairs.<n>BERT-based extraction methods require large amounts of task-specific training data.<n>This paper explores using large language models (LLMs) as a more training-data efficient and robust alternative.
  arXiv  Detail & Related papers  (2023-10-19T07:39:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.