Scene Text Retrieval via Joint Text Detection and Similarity Learning

• URL: http://arxiv.org/abs/2104.01552v1
• Date: Sun, 4 Apr 2021 07:18:38 GMT
• Title: Scene Text Retrieval via Joint Text Detection and Similarity Learning
• Authors: Hao Wang, Xiang Bai, Mingkun Yang, Shenggao Zhu, Jing Wang, Wenyu Liu
• Abstract summary: Scene text retrieval aims to localize and search all text instances from an image gallery, which are the same or similar to a given query text. We address this problem by directly learning a cross-modal similarity between a query text and each text instance from natural images. In this way, scene text retrieval can be simply performed by ranking the detected text instances with the learned similarity.
• Score: 68.24531728554892
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene text retrieval aims to localize and search all text instances from an image gallery, which are the same or similar to a given query text. Such a task is usually realized by matching a query text to the recognized words, outputted by an end-to-end scene text spotter. In this paper, we address this problem by directly learning a cross-modal similarity between a query text and each text instance from natural images. Specifically, we establish an end-to-end trainable network, jointly optimizing the procedures of scene text detection and cross-modal similarity learning. In this way, scene text retrieval can be simply performed by ranking the detected text instances with the learned similarity. Experiments on three benchmark datasets demonstrate our method consistently outperforms the state-of-the-art scene text spotting/retrieval approaches. In particular, the proposed framework of joint detection and similarity learning achieves significantly better performance than separated methods. Code is available at: https://github.com/lanfeng4659/STR-TDSL.

Related papers

• Efficiently Leveraging Linguistic Priors for Scene Text Spotting [63.22351047545888]
  This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models. We generate text distributions that align well with scene text datasets, removing the need for in-domain fine-tuning. Experimental results show that our method not only improves recognition accuracy but also enables more accurate localization of words.
  arXiv  Detail & Related papers  (2024-02-27T01:57:09Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval [66.66400551173619]
  We propose a full transformer architecture to unify cross-modal retrieval scenarios in a single $textbfVi$sion. We develop dual contrastive learning losses to embed both image-text pairs and fusion-text pairs into a common cross-modal space. Experimental results show that ViSTA outperforms other methods by at least $bf8.4%$ at Recall@1 for scene text aware retrieval task.
  arXiv  Detail & Related papers  (2022-03-31T03:40:21Z)
• Towards End-to-End Unified Scene Text Detection and Layout Analysis [60.68100769639923]
  We introduce the task of unified scene text detection and layout analysis. The first hierarchical scene text dataset is introduced to enable this novel research task. We also propose a novel method that is able to simultaneously detect scene text and form text clusters in a unified way.
  arXiv  Detail & Related papers  (2022-03-28T23:35:45Z)
• CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning [65.57338873921168]
  Localizing text instances in natural scenes is regarded as a fundamental challenge in computer vision. In this work, we quantitatively analyze the sub-text problem and present a simple yet effective design, COntrastive RElation (CORE) module. We integrate the CORE module into a two-stage text detector of Mask R-CNN and devise our text detector CORE-Text.
  arXiv  Detail & Related papers  (2021-12-14T16:22:25Z)
• Video Text Tracking With a Spatio-Temporal Complementary Model [46.99051486905713]
  Text tracking is to track multiple texts in a video,and construct a trajectory for each text. Existing methodle this task by utilizing the tracking-by-detection frame-work. We argue that the tracking accuracy of this paradigmis severely limited in more complex scenarios.
  arXiv  Detail & Related papers  (2021-11-09T08:23:06Z)
• StacMR: Scene-Text Aware Cross-Modal Retrieval [19.54677614738065]
  Cross-modal retrieval models have benefited from an increasingly rich understanding of visual scenes. Current models overlook a key aspect: the text appearing in images, which may contain crucial information for retrieval. We propose a new dataset that allows exploration of cross-modal retrieval where images contain scene-text instances.
  arXiv  Detail & Related papers  (2020-12-08T10:04:25Z)
• Textual Visual Semantic Dataset for Text Spotting [27.788077963411624]
  Text Spotting in the wild consists of detecting and recognizing text appearing in images. This is a challenging problem due to the complexity of the context where texts appear. We propose a visual context dataset for Text Spotting in the wild.
  arXiv  Detail & Related papers  (2020-04-21T23:58:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.