Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

• URL: http://arxiv.org/abs/2407.13559v1
• Date: Thu, 18 Jul 2024 14:31:09 GMT
• Title: Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition
• Authors: Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, Muhammad Abdul-Mageed,
• Abstract summary: This study introduces Qalam, a novel foundation model designed for Arabic OCR and HWR. Our model significantly outperforms existing methods, achieving a Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks.
• Score: 18.280762424107408
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) pose unique challenges due to the cursive and context-sensitive nature of the Arabic script. This study introduces Qalam, a novel foundation model designed for Arabic OCR and HWR, built on a SwinV2 encoder and RoBERTa decoder architecture. Our model significantly outperforms existing methods, achieving a Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks. We train Qalam on a diverse dataset, including over 4.5 million images from Arabic manuscripts and a synthetic dataset comprising 60k image-text pairs. Notably, Qalam demonstrates exceptional handling of Arabic diacritics, a critical feature in Arabic scripts. Furthermore, it shows a remarkable ability to process high-resolution inputs, addressing a common limitation in current OCR systems. These advancements underscore Qalam's potential as a leading solution for Arabic script recognition, offering a significant leap in accuracy and efficiency.

Related papers

• HATFormer: Historic Handwritten Arabic Text Recognition with Transformers [6.3660090769559945]
  Arabic handwriting datasets are smaller compared to English ones, making it difficult to train generalizable Arabic HTR models. We propose HATFormer, a transformer-based encoder-decoder architecture that builds on a state-of-the-art English HTR model. Our work demonstrates the feasibility of adapting an English HTR method to a low-resource language with complex, language-specific challenges.
  arXiv  Detail & Related papers  (2024-10-03T03:43:29Z)
• HierCode: A Lightweight Hierarchical Codebook for Zero-shot Chinese Text Recognition [47.86479271322264]
  We propose HierCode, a novel and lightweight codebook that exploits the innate hierarchical nature of Chinese characters. HierCode employs a multi-hot encoding strategy, leveraging hierarchical binary tree encoding and prototype learning to create distinctive, informative representations for each character. This approach not only facilitates zero-shot recognition of OOV characters by utilizing shared radicals and structures but also excels in line-level recognition tasks by computing similarity with visual features.
  arXiv  Detail & Related papers  (2024-03-20T17:20:48Z)
• Towards Codable Watermarking for Injecting Multi-bits Information to LLMs [86.86436777626959]
  Large language models (LLMs) generate texts with increasing fluency and realism. Existing watermarking methods are encoding-inefficient and cannot flexibly meet the diverse information encoding needs. We propose Codable Text Watermarking for LLMs (CTWL) that allows text watermarks to carry multi-bit customizable information.
  arXiv  Detail & Related papers  (2023-07-29T14:11:15Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• Beyond Arabic: Software for Perso-Arabic Script Manipulation [67.31374614549237]
  We provide a set of finite-state transducer (FST) components and corresponding utilities for manipulating the writing systems of languages that use the Perso-Arabic script. The library also provides simple FST-based romanization and transliteration.
  arXiv  Detail & Related papers  (2023-01-26T20:37:03Z)
• Huruf: An Application for Arabic Handwritten Character Recognition Using Deep Learning [0.0]
  We propose a lightweight Convolutional Neural Network-based architecture for recognizing Arabic characters and digits. The proposed pipeline consists of a total of 18 layers containing four layers each for convolution, pooling, batch normalization, dropout, and finally one Global average layer. The proposed model respectively achieved an accuracy of 96.93% and 99.35% which is comparable to the state-of-the-art and makes it a suitable solution for real-life end-level applications.
  arXiv  Detail & Related papers  (2022-12-16T17:39:32Z)
• Graphemic Normalization of the Perso-Arabic Script [47.429213930688086]
  This paper documents the challenges that Perso-Arabic presents beyond the best-documented languages. We focus on the situation in natural language processing (NLP), which is affected by multiple, often neglected, issues. We evaluate the effects of script normalization on eight languages from diverse language families in the Perso-Arabic script diaspora on machine translation and statistical language modeling tasks.
  arXiv  Detail & Related papers  (2022-10-21T21:59:44Z)
• Comprehensive Benchmark Datasets for Amharic Scene Text Detection and Recognition [56.048783994698425]
  Ethiopic/Amharic script is one of the oldest African writing systems, which serves at least 23 languages in East Africa. The Amharic writing system, Abugida, has 282 syllables, 15 punctuation marks, and 20 numerals. We presented the first comprehensive public datasets named HUST-ART, HUST-AST, ABE, and Tana for Amharic script detection and recognition in the natural scene.
  arXiv  Detail & Related papers  (2022-03-23T03:19:35Z)
• Arabic Handwritten Character Recognition based on Convolution Neural Networks and Support Vector Machine [0.0]
  We present an algorithm for recognizing Arabic letters and characters based on using deep convolution neural networks (DCNN) and support vector machine (SVM) This paper addresses the problem of recognizing the Arabic handwritten characters by determining the similarity between the input templates and the pre-stored templates. The experimental results of this work indicate the ability of the proposed algorithm to recognize, identify, and verify the input handwritten Arabic characters.
  arXiv  Detail & Related papers  (2020-09-28T16:18:52Z)
• An Efficient Language-Independent Multi-Font OCR for Arabic Script [0.0]
  This paper proposes a complete Arabic OCR system that takes a scanned image of Arabic Naskh script as an input and generates a corresponding digital document. This paper also proposes an improved font-independent character algorithm that outperforms the state-of-the-art segmentation algorithms.
  arXiv  Detail & Related papers  (2020-09-18T22:57:03Z)
• Neural Computing for Online Arabic Handwriting Character Recognition using Hard Stroke Features Mining [0.0]
  An enhanced method of detecting the desired critical points from vertical and horizontal direction-length of handwriting stroke features of online Arabic script recognition is proposed. A minimum feature set is extracted from these tokens for classification of characters using a multilayer perceptron with a back-propagation learning algorithm and modified sigmoid function-based activation function. The proposed method achieves an average accuracy of 98.6% comparable in state of art character recognition techniques.
  arXiv  Detail & Related papers  (2020-05-02T23:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.