Rethinking Text-Based Image Retrieval in Specific Domain

• URL: http://arxiv.org/abs/2608.10524v1
• Date: Tue, 11 Aug 2026 06:00:27 GMT
• Title: Rethinking Text-Based Image Retrieval in Specific Domain
• Abstract summary: Security Multi-Match TBIR (SecMM-TBIR) is a benchmark comprising 50k surveillance images with 200 comprehensive queries.<n>We propose the Semantic-Aware Fine-Tuning (SAFT) framework to address semantic compression in specific domains.<n>Experiments across diverse CLIP-like models demonstrate that SAFT yields an average mAP@20 gain of 7.8 points on SecMM-TBIR over standard image-text contrastive.
• Score: 12.468935521282434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Driven by the rapid advancement of vision-language representation learning, Text-based Image Retrieval (TBIR) has made notable progress. However, existing benchmarks are predominantly constructed on an exclusive single-match assumption between query and images. While effective in general scenarios, this assumption fails to reflect practical system performance in specific domains (e.g., surveillance), where a single query often corresponds to multiple relevant candidate images. To address this limitation, we design a Domain-Specific Multi-Match Text-based Image Retrieval (DSMM-TBIR) data engine. Leveraging this engine, we construct Security Multi-Match TBIR (SecMM-TBIR), a benchmark comprising 50k surveillance images with 200 comprehensive queries. Furthermore, we observe that vanilla contrastive learning in specific domains suffers from severe false negatives, forcing the model to push apart semantically similar pairs and thus degrading retrieval performance. We propose the Semantic-Aware Fine-Tuning (SAFT) framework to address semantic compression in specific domains, which incorporates Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD) to establish a promising paradigm for domain-specific TBIR tasks. Experiments across diverse CLIP-like models demonstrate that SAFT yields an average mAP@20 gain of 7.8 points on SecMM-TBIR over standard image-text contrastive (ITC) fine-tuning, while also improving general-domain performance. The entire benchmark will be released to facilitate further research.

Related papers

• UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery [69.87589293875112]
  Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets.<n>Existing referring expression comprehension benchmarks and methods are largely built around text-only queries and single-object outputs.<n>We introduce emphUniversal Referring, a generalized UAV referring task that jointly expands the query modality and the output cardinality.
  arXiv  Detail & Related papers  (2026-07-09T09:09:31Z)
• TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval [10.927254533551563]
  E-commerce image search often takes a cropped image as the query, while each candidate is represented by full item images and structured text.<n>We propose TIGER-FG, a text-guided implicit fine-grained grounding framework for e-commerce retrieval.
  arXiv  Detail & Related papers  (2026-05-18T14:07:20Z)
• TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval [61.77510161631665]
  Composed Image Retrieval (CIR) enables users to retrieve a target image using a multimodal query that consists of a reference image and modification text.<n>We construct two instruction-rich multi-modification datasets, M-FashionIQ and M-CIRR.<n>We also propose TEMA, the first CIR framework designed for multi-modification while also accommodating simple modifications.
  arXiv  Detail & Related papers  (2026-04-23T16:03:12Z)
• A Sketch+Text Composed Image Retrieval Dataset for Thangka [14.600552992453977]
  Composed Image Retrieval (CIR) enables image retrieval by combining multiple query modalities.<n>CIRThan is a sketch+text Composed Image Retrieval dataset for Thangka imagery.
  arXiv  Detail & Related papers  (2026-02-09T09:14:29Z)
• Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models [58.46663983451155]
  PixSearch is an end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning.<n>During encoding, PixSearch emits search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries.<n>On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization.
  arXiv  Detail & Related papers  (2026-01-27T00:46:08Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation [15.941958367737408]
  Seg-TTO is a framework for zero-shot, open-vocabulary semantic segmentation.<n>We focus on segmentation-specific test-time optimization to address this gap.<n>Seg-TTO demonstrates clear performance improvements (up to 27% mIoU increase on some datasets) establishing new state-of-the-art.
  arXiv  Detail & Related papers  (2025-01-08T18:58:24Z)
• EFSA: Episodic Few-Shot Adaptation for Text-to-Image Retrieval [4.542714912924457]
  We introduce Episodic Few-Shot Adaptation (EFSA), a novel test-time framework that adapts pre-trained models dynamically to a query's domain.<n>EFSA improves performance across diverse domains while preserving generalization.<n>Our work highlights the potential of episodic few-shot adaptation to enhance robustness in the critical and understudied task of open-domain text-to-image retrieval.
  arXiv  Detail & Related papers  (2024-11-28T17:09:20Z)
• Cross-Domain Few-Shot Semantic Segmentation via Doubly Matching Transformation [26.788260801305974]
  Cross-Domain Few-shot Semantic (CD-FSS) aims to train generalized models that can segment classes from different domains with a few labeled images. Previous works have proven the effectiveness of feature transformation in addressing CD-FSS. We propose a Doubly Matching Transformation-based Network (DMTNet) to solve the above issue.
  arXiv  Detail & Related papers  (2024-05-24T06:47:43Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Disentangled Feature Representation for Few-shot Image Classification [64.40410801469106]
  We propose a novel Disentangled Feature Representation framework, dubbed DFR, for few-shot learning applications. DFR can adaptively decouple the discriminative features that are modeled by the classification branch, from the class-irrelevant component of the variation branch. In general, most of the popular deep few-shot learning methods can be plugged in as the classification branch, thus DFR can boost their performance on various few-shot tasks.
  arXiv  Detail & Related papers  (2021-09-26T09:53:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.