Logographic Character Visual Pretraining via Semantic-based Contrastive Learning

• URL: http://arxiv.org/abs/2608.00096v1
• Date: Thu, 30 Jul 2026 17:12:18 GMT
• Title: Logographic Character Visual Pretraining via Semantic-based Contrastive Learning
• Abstract summary: We propose a novel method for logographic character recognition, which introduces a multi-modal learning approach using visual semantics and contextual semantics of characters.<n>A novel pre-training strategy is designed to enhance deep visual representations, especially for datasets suffering from issues of imbalanced and rare instances.
• Score: 11.660518219984107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current deep learning-based character vision studies, e.g., text recognition, character image denoising, and historical text completion, are offering new solutions for learning, managing, and utilizing character resources. However, the performance of these studies peaks only with large and balanced datasets, which is a rarity with real-world character datasets, especially for logographic character languages, e.g., Chinese. The imbalance in data distribution of logographic characters is a common issue due to differences in character usage frequency and new characters being continuously created. In this paper, we propose a novel method for logographic character recognition, which introduces a multi-modal learning approach using visual semantics and contextual semantics of characters. A novel pre-training strategy is designed to enhance deep visual representations, especially for datasets suffering from issues of imbalanced and rare instances, by extracting the contextual semantics of each character from the corresponding language models. We conduct experiments across various datasets to evaluate our character recognition method and further validate the contrastive pre-training strategy by several downstream tasks. Experimental results demonstrate the superiority of our method compared to state-of-the-art methods.

Related papers

• Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing [71.29488677105127]
  Existing scene text recognition (STR) methods struggle to recognize challenging texts, especially for artistic and severely distorted characters. We propose a contrastive learning-based STR framework by leveraging synthetic and real unlabeled data without any human cost. Our method achieves SOTA performance (94.7% and 70.9% average accuracy on common benchmarks and Union14M-Benchmark.
  arXiv  Detail & Related papers  (2024-11-23T15:24:47Z)
• Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning [35.47078178526536]
  Recent advancements in pre-trained large-scale language-image models have ushered in a new era of visual comprehension. This paper tackles two well-known issues within the realm of visual analytics: (1) the efficient exploration of large-scale image datasets and identification of potential data biases within them; (2) the evaluation of image captions and steering of their generation process.
  arXiv  Detail & Related papers  (2023-11-02T06:21:35Z)
• Multi-level Contrastive Learning for Script-based Character Understanding [14.341307979533871]
  We tackle the scenario of understanding characters in scripts, which aims to learn the characters' personalities and identities from their utterances. We propose a multi-level contrastive learning framework to capture characters' global information in a fine-grained manner.
  arXiv  Detail & Related papers  (2023-10-20T02:40:52Z)
• Text-Only Training for Visual Storytelling [107.19873669536523]
  We formulate visual storytelling as a visual-conditioned story generation problem. We propose a text-only training method that separates the learning of cross-modality alignment and story generation.
  arXiv  Detail & Related papers  (2023-08-17T09:32:17Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• Improving Scene Text Recognition for Character-Level Long-Tailed Distribution [35.14058653707104]
  We propose a novel Context-Aware and Free Experts Network (CAFE-Net) using two experts. CAFE-Net improves the STR performance on languages containing numerous number of characters.
  arXiv  Detail & Related papers  (2023-03-31T06:11:33Z)
• RZCR: Zero-shot Character Recognition via Radical-based Reasoning [17.305603529254608]
  RZCR consists of a visual semantic fusion-based radical information extractor (RIE) and a knowledge graph character reasoner (KGR) RZCR shows promising experimental results, especially on few-sample character datasets.
  arXiv  Detail & Related papers  (2022-07-12T21:12:05Z)
• Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition [101.60244147302197]
  We introduce contrastive learning and masked image modeling to learn discrimination and generation of text images. Our method outperforms previous self-supervised text recognition methods by 10.2%-20.2% on irregular scene text recognition datasets. Our proposed text recognizer exceeds previous state-of-the-art text recognition methods by averagely 5.3% on 11 benchmarks, with similar model size.
  arXiv  Detail & Related papers  (2022-07-01T03:50:26Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Spectral Graph-based Features for Recognition of Handwritten Characters: A Case Study on Handwritten Devanagari Numerals [0.0]
  We propose an approach that exploits the robust graph representation and spectral graph embedding concept to represent handwritten characters. For corroboration of the efficacy of the proposed method, extensive experiments were carried out on the standard handwritten numeral Computer Vision Pattern Recognition, Unit of Indian Statistical Institute Kolkata dataset.
  arXiv  Detail & Related papers  (2020-07-07T08:40:08Z)
• Separating Content from Style Using Adversarial Learning for Recognizing Text in the Wild [103.51604161298512]
  We propose an adversarial learning framework for the generation and recognition of multiple characters in an image. Our framework can be integrated into recent recognition methods to achieve new state-of-the-art recognition accuracy.
  arXiv  Detail & Related papers  (2020-01-13T12:41:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.