Region Prompt Tuning: Fine-grained Scene Text Detection Utilizing Region Text Prompt

• URL: http://arxiv.org/abs/2409.13576v2
• Date: Tue, 19 Nov 2024 19:26:07 GMT
• Title: Region Prompt Tuning: Fine-grained Scene Text Detection Utilizing Region Text Prompt
• Authors: Xingtao Lin, Heqian Qiu, Lanxiao Wang, Ruihang Wang, Linfeng Xu, Hongliang Li,
• Abstract summary: Region prompt tuning method decomposes region text prompt into individual characters and splits visual feature map into region visual tokens. This allows a character matches the local features of a token, thereby avoiding the omission of detailed features and fine-grained text. Our proposed method combines a general score map from the image-text process with a region score map derived from character-token matching.
• Score: 10.17947324152468
• License:
• Abstract: Recent advancements in prompt tuning have successfully adapted large-scale models like Contrastive Language-Image Pre-trained (CLIP) for downstream tasks such as scene text detection. Typically, text prompt complements the text encoder's input, focusing on global features while neglecting fine-grained details, leading to fine-grained text being ignored in task of scene text detection. In this paper, we propose the region prompt tuning (RPT) method for fine-grained scene text detection, where region text prompt proposed would help focus on fine-grained features. Region prompt tuning method decomposes region text prompt into individual characters and splits visual feature map into region visual tokens, creating a one-to-one correspondence between characters and tokens. This allows a character matches the local features of a token, thereby avoiding the omission of detailed features and fine-grained text. To achieve this, we introduce a sharing position embedding to link each character with its corresponding token and employ a bidirectional distance loss to align each region text prompt character with the target ``text''. To refine the information at fine-grained level, we implement character-token level interactions before and after encoding. Our proposed method combines a general score map from the image-text process with a region score map derived from character-token matching, producing a final score map that could balance the global and local features and be fed into DBNet to detect the text. Experiments on benchmarks like ICDAR2015, TotalText, and CTW1500 demonstrate RPT impressive performance, underscoring its effectiveness for scene text detection.

Related papers

• Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis [52.34110239735265]
  We present Text Grouping Adapter (TGA), a module that can enable the utilization of various pre-trained text detectors to learn layout analysis. Our comprehensive experiments demonstrate that, even with frozen pre-trained models, incorporating our TGA into various pre-trained text detectors and text spotters can achieve superior layout analysis performance.
  arXiv  Detail & Related papers  (2024-05-13T05:48:35Z)
• Text Region Multiple Information Perception Network for Scene Text Detection [19.574306663095243]
  This paper proposes a plug-and-play module called the Region Multiple Information Perception Module (RMIPM) to enhance the detection performance of segmentation-based algorithms. Specifically, we design an improved module that can perceive various types of information about scene text regions, such as text foreground classification maps, distance maps, direction maps, etc.
  arXiv  Detail & Related papers  (2024-01-18T14:36:51Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Expressive Text-to-Image Generation with Rich Text [42.923053338525804]
  We propose a rich-text editor supporting formats such as font style, size, color, and footnote. We extract each word's attributes from rich text to enable local style control, explicit token reweighting, precise color rendering, and detailed region synthesis.
  arXiv  Detail & Related papers  (2023-04-13T17:59:55Z)
• Contextual Text Block Detection towards Scene Text Understanding [85.40898487745272]
  This paper presents contextual text detection, a new setup that detects contextual text blocks (CTBs) for better understanding of texts in scenes. We formulate the new setup by a dual detection task which first detects integral text units and then groups them into a CTB. To this end, we design a novel scene text clustering technique that treats integral text units as tokens and groups them (belonging to the same CTB) into an ordered token sequence.
  arXiv  Detail & Related papers  (2022-07-26T14:59:25Z)
• Text Detection & Recognition in the Wild for Robot Localization [1.52292571922932]
  We propose an end-to-end scene text spotting model that simultaneously outputs the text string and bounding boxes. Our central contribution is introducing utilizing an end-to-end scene text spotting framework to adequately capture the irregular and occluded text regions.
  arXiv  Detail & Related papers  (2022-05-17T18:16:34Z)
• Scene Text Retrieval via Joint Text Detection and Similarity Learning [68.24531728554892]
  Scene text retrieval aims to localize and search all text instances from an image gallery, which are the same or similar to a given query text. We address this problem by directly learning a cross-modal similarity between a query text and each text instance from natural images. In this way, scene text retrieval can be simply performed by ranking the detected text instances with the learned similarity.
  arXiv  Detail & Related papers  (2021-04-04T07:18:38Z)
• MANGO: A Mask Attention Guided One-Stage Scene Text Spotter [41.66707532607276]
  We propose a novel Mask AttentioN Guided One-stage text spotting framework named MANGO. The proposed method achieves competitive and even new state-of-the-art performance on both regular and irregular text spotting benchmarks.
  arXiv  Detail & Related papers  (2020-12-08T10:47:49Z)
• Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting [49.768327669098674]
  We propose an end-to-end trainable text spotting approach named Text Perceptron. It first employs an efficient segmentation-based text detector that learns the latent text reading order and boundary information. Then a novel Shape Transform Module (abbr. STM) is designed to transform the detected feature regions into regular morphologies.
  arXiv  Detail & Related papers  (2020-02-17T08:07:19Z)
• TextScanner: Reading Characters in Order for Robust Scene Text Recognition [60.04267660533966]
  TextScanner is an alternative approach for scene text recognition. It generates pixel-wise, multi-channel segmentation maps for character class, position and order. It also adopts RNN for context modeling and performs paralleled prediction for character position and class.
  arXiv  Detail & Related papers  (2019-12-28T07:52:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.