Show Me the World in My Language: Establishing the First Baseline for Scene-Text to Scene-Text Translation

• URL: http://arxiv.org/abs/2308.03024v3
• Date: Mon, 2 Sep 2024 05:51:02 GMT
• Title: Show Me the World in My Language: Establishing the First Baseline for Scene-Text to Scene-Text Translation
• Authors: Shreyas Vaidya, Arvind Kumar Sharma, Prajwal Gatti, Anand Mishra,
• Abstract summary: We study the task of visually'' translating scene text from a source language to a target language. Visual translation involves not just the recognition and translation of scene text but also the generation of the translated image. We present a cascaded framework for visual translation that combines state-of-the-art modules for scene text recognition, machine translation, and scene text synthesis.
• Score: 1.9085074258303771
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this work, we study the task of ``visually'' translating scene text from a source language (e.g., Hindi) to a target language (e.g., English). Visual translation involves not just the recognition and translation of scene text but also the generation of the translated image that preserves visual features of the source scene text, such as font, size, and background. There are several challenges associated with this task, such as translation with limited context, deciding between translation and transliteration, accommodating varying text lengths within fixed spatial boundaries, and preserving the font and background styles of the source scene text in the target language. To address this problem, we make the following contributions: (i) We study visual translation as a standalone problem for the first time in the literature. (ii) We present a cascaded framework for visual translation that combines state-of-the-art modules for scene text recognition, machine translation, and scene text synthesis as a baseline for the task. (iii) We propose a set of task-specific design enhancements to design a variant of the baseline to obtain performance improvements. (iv) Currently, the existing related literature lacks any comprehensive performance evaluation for this novel task. To fill this gap, we introduce several automatic and user-assisted evaluation metrics designed explicitly for evaluating visual translation. Further, we evaluate presented baselines for translating scene text between Hindi and English. Our experiments demonstrate that although we can effectively perform visual translation over a large collection of scene text images, the presented baseline only partially addresses challenges posed by visual translation tasks. We firmly believe that this new task and the limitations of existing models, as reported in this paper, should encourage further research in visual translation.

Related papers

• Towards Visual Text Design Transfer Across Languages [49.78504488452978]
  We introduce a novel task of Multimodal Style Translation (MuST-Bench) MuST-Bench is a benchmark designed to evaluate the ability of visual text generation models to perform translation across different writing systems. In response, we introduce SIGIL, a framework for multimodal style translation that eliminates the need for style descriptions.
  arXiv  Detail & Related papers  (2024-10-24T15:15:01Z)
• ViConsFormer: Constituting Meaningful Phrases of Scene Texts using Transformer-based Method in Vietnamese Text-based Visual Question Answering [0.5803309695504829]
  Main challenge of text-based VQA is exploiting the meaning and information from scene texts. Recent studies tackled this challenge by considering the spatial information of scene texts in images. We introduce a novel method that effectively exploits the information from scene texts written in Vietnamese.
  arXiv  Detail & Related papers  (2024-10-18T03:00:03Z)
• AnyTrans: Translate AnyText in the Image with Large Scale Models [88.5887934499388]
  This paper introduces AnyTrans, an all-encompassing framework for the task-Translate AnyText in the Image (TATI) Our framework incorporates contextual cues from both textual and visual elements during translation. We have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.
  arXiv  Detail & Related papers  (2024-06-17T11:37:48Z)
• Dataset and Benchmark for Urdu Natural Scenes Text Detection, Recognition and Visual Question Answering [50.52792174648067]
  This initiative seeks to bridge the gap between textual and visual comprehension. We propose a new multi-task Urdu scene text dataset comprising over 1000 natural scene images. We provide fine-grained annotations for text instances, addressing the limitations of previous datasets.
  arXiv  Detail & Related papers  (2024-05-21T06:48:26Z)
• The First Swahili Language Scene Text Detection and Recognition Dataset [55.83178123785643]
  There is a significant gap in low-resource languages, especially the Swahili Language. Swahili is widely spoken in East African countries but is still an under-explored language in scene text recognition. We propose a comprehensive dataset of Swahili scene text images and evaluate the dataset on different scene text detection and recognition models.
  arXiv  Detail & Related papers  (2024-05-19T03:55:02Z)
• Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model [31.819060415422353]
  Diff-Text is a training-free scene text generation framework for any language. Our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.
  arXiv  Detail & Related papers  (2023-12-19T15:18:40Z)
• FASTER: A Font-Agnostic Scene Text Editing and Rendering Framework [19.564048493848272]
  Scene Text Editing (STE) is a challenging research problem, that primarily aims towards modifying existing texts in an image. Existing style-transfer-based approaches have shown sub-par editing performance due to complex image backgrounds, diverse font attributes, and varying word lengths within the text. We propose a novel font-agnostic scene text editing and rendering framework, named FASTER, for simultaneously generating text in arbitrary styles and locations.
  arXiv  Detail & Related papers  (2023-08-05T15:54:06Z)
• Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination [88.74459704391214]
  In this work, we investigate a more realistic unsupervised multimodal machine translation (UMMT) setup. We represent the input images and texts with the visual and language scene graphs (SG), where such fine-grained vision-language features ensure a holistic understanding of the semantics. Several SG-pivoting based learning objectives are introduced for unsupervised translation training. Our method outperforms the best-performing baseline by significant BLEU scores on the task and setup.
  arXiv  Detail & Related papers  (2023-05-20T18:17:20Z)
• Augmented Transformers with Adaptive n-grams Embedding for Multilingual Scene Text Recognition [10.130342722193204]
  This paper proposes an augmented transformer architecture with n-grams embedding and cross-language rectification (TANGER) TANGER consists of a primary transformer with single patch embeddings of visual images, and a supplementary transformer with adaptive n-grams embeddings. Cross-language rectification is achieved with a loss function that takes into account both language identification and contextual coherence scoring.
  arXiv  Detail & Related papers  (2023-02-28T02:37:30Z)
• ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval [66.66400551173619]
  We propose a full transformer architecture to unify cross-modal retrieval scenarios in a single $textbfVi$sion. We develop dual contrastive learning losses to embed both image-text pairs and fusion-text pairs into a common cross-modal space. Experimental results show that ViSTA outperforms other methods by at least $bf8.4%$ at Recall@1 for scene text aware retrieval task.
  arXiv  Detail & Related papers  (2022-03-31T03:40:21Z)
• Simultaneous Machine Translation with Visual Context [42.88121241096681]
  Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible. We analyse the impact of different multimodal approaches and visual features on state-of-the-art SiMT frameworks.
  arXiv  Detail & Related papers  (2020-09-15T18:19:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.