Towards Fully Automated Manga Translation
- URL: http://arxiv.org/abs/2012.14271v3
- Date: Sat, 9 Jan 2021 14:21:31 GMT
- Title: Towards Fully Automated Manga Translation
- Authors: Ryota Hinami, Shonosuke Ishiwatari, Kazuhiko Yasuda, and Yusuke Matsui
- Abstract summary: We tackle the problem of machine translation of manga, Japanese comics.
obtaining context from the image is essential for manga translation.
First, we propose multimodal context-aware translation framework.
Second, for training the model, we propose the approach to automatic corpus construction from pairs of original manga.
Third, we created a new benchmark to evaluate manga translation.
- Score: 8.45043706496877
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We tackle the problem of machine translation of manga, Japanese comics. Manga
translation involves two important problems in machine translation:
context-aware and multimodal translation. Since text and images are mixed up in
an unstructured fashion in Manga, obtaining context from the image is essential
for manga translation. However, it is still an open problem how to extract
context from image and integrate into MT models. In addition, corpus and
benchmarks to train and evaluate such model is currently unavailable. In this
paper, we make the following four contributions that establishes the foundation
of manga translation research. First, we propose multimodal context-aware
translation framework. We are the first to incorporate context information
obtained from manga image. It enables us to translate texts in speech bubbles
that cannot be translated without using context information (e.g., texts in
other speech bubbles, gender of speakers, etc.). Second, for training the
model, we propose the approach to automatic corpus construction from pairs of
original manga and their translations, by which large parallel corpus can be
constructed without any manual labeling. Third, we created a new benchmark to
evaluate manga translation. Finally, on top of our proposed methods, we devised
a first comprehensive system for fully automated manga translation.
Related papers
- Context-Informed Machine Translation of Manga using Multimodal Large Language Models [4.063595992745368]
We investigate what extent multimodal large language models (LLMs) can provide effective manga translation.
Specifically, we propose a methodology that leverages the vision component of multimodal LLMs to improve translation quality.
We introduce a new evaluation dataset -- the first parallel Japanese-Polish manga translation dataset.
arXiv Detail & Related papers (2024-11-04T20:29:35Z) - AnyTrans: Translate AnyText in the Image with Large Scale Models [88.5887934499388]
This paper introduces AnyTrans, an all-encompassing framework for the task-Translate AnyText in the Image (TATI)
Our framework incorporates contextual cues from both textual and visual elements during translation.
We have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.
arXiv Detail & Related papers (2024-06-17T11:37:48Z) - An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance [53.974497865647336]
We take a first step towards translating images to make them culturally relevant.
We build three pipelines comprising state-of-the-art generative models to do the task.
We conduct a human evaluation of translated images to assess for cultural relevance and meaning preservation.
arXiv Detail & Related papers (2024-04-01T17:08:50Z) - The Manga Whisperer: Automatically Generating Transcriptions for Comics [55.544015596503726]
We present a unified model, Magi, that is able to detect panels, text boxes and character boxes.
We propose a novel approach that is able to sort the detected text boxes in their reading order and generate a dialogue transcript.
arXiv Detail & Related papers (2024-01-18T18:59:09Z) - M2C: Towards Automatic Multimodal Manga Complement [40.01354682367365]
Multimodal manga analysis focuses on enhancing manga understanding with visual and textual features.
Currently, most comics are hand-drawn and prone to problems such as missing pages, text contamination, and aging.
We first propose the Multimodal Manga Complement task by establishing a new M2C benchmark dataset covering two languages.
arXiv Detail & Related papers (2023-10-26T04:10:16Z) - Dense Multitask Learning to Reconfigure Comics [63.367664789203936]
We develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels.
Our method can successfully identify the semantic units as well as the notion of 3D in comic panels.
arXiv Detail & Related papers (2023-07-16T15:10:34Z) - A Bilingual Parallel Corpus with Discourse Annotations [82.07304301996562]
This paper describes BWB, a large parallel corpus first introduced in Jiang et al. (2022), along with an annotated test set.
The BWB corpus consists of Chinese novels translated by experts into English, and the annotated test set is designed to probe the ability of machine translation systems to model various discourse phenomena.
arXiv Detail & Related papers (2022-10-26T12:33:53Z) - BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine
Translation [53.55009917938002]
We propose to refine the mined bitexts via automatic editing.
Experiments demonstrate that our approach successfully improves the quality of CCMatrix mined bitext for 5 low-resource language-pairs and 10 translation directions by up to 8 BLEU points.
arXiv Detail & Related papers (2021-11-12T16:00:39Z) - Simultaneous Machine Translation with Visual Context [42.88121241096681]
Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible.
We analyse the impact of different multimodal approaches and visual features on state-of-the-art SiMT frameworks.
arXiv Detail & Related papers (2020-09-15T18:19:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.