Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts

• URL: http://arxiv.org/abs/2506.04999v1
• Date: Thu, 05 Jun 2025 13:10:17 GMT
• Title: Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
• Authors: Gengluo Li, Huawen Shen, Yu Zhou,
• Abstract summary: Chinese scene text retrieval is a practical task that aims to search for images containing visual instances of a Chinese query text.<n>Current efforts tend to inherit the solution for English scene text retrieval, failing to achieve satisfactory performance.<n>We propose Chinese Scene Text Retrieval CLIP (CSTR-CLIP), a novel model that integrates global visual information with multi-granularity alignment training.
• Score: 2.9100667158464035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Chinese scene text retrieval is a practical task that aims to search for images containing visual instances of a Chinese query text. This task is extremely challenging because Chinese text often features complex and diverse layouts in real-world scenes. Current efforts tend to inherit the solution for English scene text retrieval, failing to achieve satisfactory performance. In this paper, we establish a Diversified Layout benchmark for Chinese Street View Text Retrieval (DL-CSVTR), which is specifically designed to evaluate retrieval performance across various text layouts, including vertical, cross-line, and partial alignments. To address the limitations in existing methods, we propose Chinese Scene Text Retrieval CLIP (CSTR-CLIP), a novel model that integrates global visual information with multi-granularity alignment training. CSTR-CLIP applies a two-stage training process to overcome previous limitations, such as the exclusion of visual features outside the text region and reliance on single-granularity alignment, thereby enabling the model to effectively handle diverse text layouts. Experiments on existing benchmark show that CSTR-CLIP outperforms the previous state-of-the-art model by 18.82% accuracy and also provides faster inference speed. Further analysis on DL-CSVTR confirms the superior performance of CSTR-CLIP in handling various text layouts. The dataset and code will be publicly available to facilitate research in Chinese scene text retrieval.

Related papers

• Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval [13.315951821189538]
  Scene text retrieval aims to find all images containing the query text from an image gallery. Current efforts tend to adopt an Optical Character Recognition (OCR) pipeline, which requires complicated text detection and/or recognition processes. We propose to explore the intrinsic potential of Contrastive Language-Image Pre-training (CLIP) for OCR-free scene text retrieval.
  arXiv  Detail & Related papers  (2024-08-01T10:25:14Z)
• Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs [96.54224331778195]
  We present a text-grounding document understanding model, termed TGDoc, which enhances MLLMs with the ability to discern the spatial positioning of text within images. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-11-22T06:46:37Z)
• STEP -- Towards Structured Scene-Text Spotting [9.339184886724812]
  We introduce the structured scene-text spotting task, which requires a scene-text OCR system to spot text in the wild according to a query regular expression. We propose the Structured TExt sPotter (STEP), a model that exploits the provided text structure to guide the OCR process. Our approach enables accurate zero-shot structured text spotting in a wide variety of real-world reading scenarios.
  arXiv  Detail & Related papers  (2023-09-05T16:11:54Z)
• Orientation-Independent Chinese Text Recognition in Scene Images [61.34060587461462]
  We take the first attempt to extract orientation-independent visual features by disentangling content and orientation information of text images. Specifically, we introduce a Character Image Reconstruction Network (CIRN) to recover corresponding printed character images with disentangled content and orientation information.
  arXiv  Detail & Related papers  (2023-09-03T05:30:21Z)
• A Benchmark for Chinese-English Scene Text Image Super-resolution [15.042152725255171]
  Scene Text Image Super-resolution (STISR) aims to recover high-resolution (HR) scene text images with visually pleasant and readable text content from low-resolution (LR) input. Most existing works focus on recovering English texts, which have relatively simple character structures. We propose a real-world Chinese-English benchmark dataset, namely Real-CE, for the task of STISR.
  arXiv  Detail & Related papers  (2023-08-07T02:57:48Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Towards End-to-End Unified Scene Text Detection and Layout Analysis [60.68100769639923]
  We introduce the task of unified scene text detection and layout analysis. The first hierarchical scene text dataset is introduced to enable this novel research task. We also propose a novel method that is able to simultaneously detect scene text and form text clusters in a unified way.
  arXiv  Detail & Related papers  (2022-03-28T23:35:45Z)
• CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning [65.57338873921168]
  Localizing text instances in natural scenes is regarded as a fundamental challenge in computer vision. In this work, we quantitatively analyze the sub-text problem and present a simple yet effective design, COntrastive RElation (CORE) module. We integrate the CORE module into a two-stage text detector of Mask R-CNN and devise our text detector CORE-Text.
  arXiv  Detail & Related papers  (2021-12-14T16:22:25Z)
• Scene Text Retrieval via Joint Text Detection and Similarity Learning [68.24531728554892]
  Scene text retrieval aims to localize and search all text instances from an image gallery, which are the same or similar to a given query text. We address this problem by directly learning a cross-modal similarity between a query text and each text instance from natural images. In this way, scene text retrieval can be simply performed by ranking the detected text instances with the learned similarity.
  arXiv  Detail & Related papers  (2021-04-04T07:18:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.