I2C2W: Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition

• URL: http://arxiv.org/abs/2105.08383v1
• Date: Tue, 18 May 2021 09:20:58 GMT
• Title: I2C2W: Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition
• Authors: Chuhui Xue, Shijian Lu, Song Bai, Wenqing Zhang, Changhu Wang
• Abstract summary: This paper presents I2C2W, a novel scene text recognizer that is accurate and tolerant to various noises in scenes. I2C2W consists of an image-to-character module (I2C) and a character-to-word module (C2W) which are complementary and can be trained end-to-end.
• Score: 68.95544645458882
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Leveraging the advances of natural language processing, most recent scene text recognizers adopt an encoder-decoder architecture where text images are first converted to representative features and then a sequence of characters via `direct decoding'. However, scene text images suffer from rich noises of different sources such as complex background and geometric distortions which often confuse the decoder and lead to incorrect alignment of visual features at noisy decoding time steps. This paper presents I2C2W, a novel scene text recognizer that is accurate and tolerant to various noises in scenes. I2C2W consists of an image-to-character module (I2C) and a character-to-word module (C2W) which are complementary and can be trained end-to-end. I2C detects characters and predicts their relative positions in a word. It strives to detect all possible characters including incorrect and redundant ones based on different alignments of visual features without the restriction of time steps. Taking the detected characters as input, C2W learns from character semantics and their positions to filter out incorrect and redundant detection and produce the final word recognition. Extensive experiments over seven public datasets show that I2C2W achieves superior recognition performances and outperforms the state-of-the-art by large margins on challenging irregular scene text datasets.

Related papers

• UNIT: Unifying Image and Text Recognition in One Vision Encoder [51.140564856352825]
  UNIT is a novel training framework aimed at UNifying Image and Text recognition within a single model. We show that UNIT significantly outperforms existing methods on document-related tasks. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment.
  arXiv  Detail & Related papers  (2024-09-06T08:02:43Z)
• VCR: Visual Caption Restoration [80.24176572093512]
  We introduce Visual Caption Restoration (VCR), a vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images.
  arXiv  Detail & Related papers  (2024-06-10T16:58:48Z)
• Choose What You Need: Disentangled Representation Learning for Scene Text Recognition, Removal and Editing [47.421888361871254]
  Scene text images contain not only style information (font, background) but also content information (character, texture) Previous representation learning methods use tightly coupled features for all tasks, resulting in sub-optimal performance. We propose a Disentangled Representation Learning framework (DARLING) aimed at disentangling these two types of features for improved adaptability.
  arXiv  Detail & Related papers  (2024-05-07T15:00:11Z)
• Story Visualization by Online Text Augmentation with Context Memory [64.86944645907771]
  We propose a novel memory architecture for the Bi-directional Transformer framework with an online text augmentation. The proposed method significantly outperforms the state of the arts in various metrics including FID, character F1, frame accuracy, BLEU-2/3, and R-precision.
  arXiv  Detail & Related papers  (2023-08-15T05:08:12Z)
• CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels [28.42405456691034]
  We propose a two-stage strategy to facilitate a better visual representation in image re-identification tasks. The key idea is to fully exploit the cross-modal description ability in CLIP through a set of learnable text tokens for each ID. The effectiveness of the proposed strategy is validated on several datasets for the person or vehicle ReID tasks.
  arXiv  Detail & Related papers  (2022-11-25T09:41:57Z)
• Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition [63.6608759501803]
  We propose to recognize artistic text at three levels. corner points are applied to guide the extraction of local features inside characters, considering the robustness of corner structures to appearance and shape. Secondly, we design a character contrastive loss to model the character-level feature, improving the feature representation for character classification. Thirdly, we utilize Transformer to learn the global feature on image-level and model the global relationship of the corner points.
  arXiv  Detail & Related papers  (2022-07-31T14:11:05Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)
• Primitive Representation Learning for Scene Text Recognition [7.818765015637802]
  We propose a primitive representation learning method that aims to exploit intrinsic representations of scene text images. A Primitive REpresentation learning Network (PREN) is constructed to use the visual text representations for parallel decoding. We also propose a framework called PREN2D to alleviate the misalignment problem in attention-based methods.
  arXiv  Detail & Related papers  (2021-05-10T11:54:49Z)
• Text to Image Generation with Semantic-Spatial Aware GAN [41.73685713621705]
  A text to image generation (T2I) model aims to generate photo-realistic images which are semantically consistent with the text descriptions. We propose a novel framework Semantic-Spatial Aware GAN, which is trained in an end-to-end fashion so that the text encoder can exploit better text information.
  arXiv  Detail & Related papers  (2021-04-01T15:48:01Z)
• SCATTER: Selective Context Attentional Scene Text Recognizer [16.311256552979835]
  Scene Text Recognition (STR) is the task of recognizing text against complex image backgrounds. Current state-of-the-art (SOTA) methods still struggle to recognize text written in arbitrary shapes. We introduce a novel architecture for STR, named Selective Context ATtentional Text Recognizer (SCATTER)
  arXiv  Detail & Related papers  (2020-03-25T09:20:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.