IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

• URL: http://arxiv.org/abs/2603.10495v1
• Date: Wed, 11 Mar 2026 07:42:42 GMT
• Title: IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation
• Abstract summary: End-to-end In-Image Machine Translation aims to convert text embedded within an image into a target language.<n>Existing IIMT benchmarks are largely synthetic and fail to reflect real-world complexity.<n>We present IMTBench, a new benchmark of 2,500 image translation samples covering four practical scenarios and nine languages.
• Score: 19.168097777455177
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: End-to-end In-Image Machine Translation (IIMT) aims to convert text embedded within an image into a target language while preserving the original visual context, layout, and rendering style. However, existing IIMT benchmarks are largely synthetic and thus fail to reflect real-world complexity, while current evaluation protocols focus on single-modality metrics and overlook cross-modal faithfulness between rendered text and model outputs. To address these shortcomings, we present In-image Machine Translation Benchmark (IMTBench), a new benchmark of 2,500 image translation samples covering four practical scenarios and nine languages. IMTBench supports multi-aspect evaluation, including translation quality, background preservation, overall image quality, and a cross-modal alignment score that measures consistency between the translated text produced by the model and the text rendered in the translated image. We benchmark strong commercial cascade systems, and both closed- and open-source unified multi-modal models, and observe large performance gaps across scenarios and languages, especially on natural scenes and resource-limited languages, highlighting substantial headroom for end-to-end image text translation. We hope IMTBench establishes a standardized benchmark to accelerate progress in this emerging task.

Related papers

• UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation [23.787128107000374]
  In-Image Machine Translation (IIMT) aims to translate scene text in an image and render the translated text back into the original regions while preserving the overall visual appearance.<n>Recent unified multimodal models provide a promising solution by combining visual-text understanding and image generation within a single framework.<n>We present UniTranslator, a unified multimodal framework for IIMT that tightly couples translation understanding and text editing.
  arXiv  Detail & Related papers  (2026-06-23T09:11:24Z)
• MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation [29.093072408848467]
  We present MMTIT-Bench, a human-verified multilingual and multi-scenario benchmark with 1,400 images spanning fourteen non-English and non-Chinese languages.<n>Beyond benchmarking, we study how reasoning-oriented data design improves translation.<n>We propose Cognition-Perception-Reasoning for Translation (CPR-Trans), a data paradigm that integrates scene cognition, text perception, and translation reasoning within a unified reasoning process.
  arXiv  Detail & Related papers  (2026-03-25T03:37:12Z)
• PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Models [32.38746546500033]
  Text Image Machine Translation (TIMT) aims to translate texts embedded within an image into another language.<n>We extend traditional TIMT into position-aware TIMT (PATIMT), aiming to support fine-grained and layout-preserving translation.<n>We construct the PATIMT benchmark (PATIMTBench), which consists of 10 diverse real-world scenarios.<n>Specifically, we introduce an Adaptive Image OCR Refinement Pipeline, which adaptively selects appropriate OCR tools based on scenario.
  arXiv  Detail & Related papers  (2025-09-14T08:33:23Z)
• PRIM: Towards Practical In-Image Multilingual Machine Translation [48.357528732061105]
  In-Image Machine Translation (IIMT) aims to translate images containing texts from one language to another.<n>Current research of end-to-end IIMT conducts on synthetic data, with simple background, single font, fixed text position, and bilingual translation.<n>We propose an end-to-end model VisTrans to handle the challenge of practical conditions in PRIM.
  arXiv  Detail & Related papers  (2025-09-05T14:38:07Z)
• Ensuring Consistency for In-Image Translation [47.1986912570945]
  The in-image machine translation task involves translating text embedded within images, with the translated results presented in image format.<n>We propose the need to uphold two types of consistency in this task: translation consistency and image generation consistency.<n>We introduce a novel two-stage framework named HCIIT which involves text-image translation using a multimodal multilingual large language model in the first stage and image backfilling with a diffusion model in the second stage.
  arXiv  Detail & Related papers  (2024-12-24T03:50:03Z)
• Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation [81.45400849638347]
  In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this paper, we propose an end-to-end IIMT model consisting of four modules. Our model achieves competitive performance compared to cascaded models with only 70.9% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.
  arXiv  Detail & Related papers  (2024-07-03T08:15:39Z)
• AnyTrans: Translate AnyText in the Image with Large Scale Models [88.5887934499388]
  This paper introduces AnyTrans, an all-encompassing framework for the task-Translate AnyText in the Image (TATI) Our framework incorporates contextual cues from both textual and visual elements during translation. We have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.
  arXiv  Detail & Related papers  (2024-06-17T11:37:48Z)
• Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts [107.32683485639654]
  Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set. One such benchmark, "Conceptual Coverage Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I models by prompting them to generate pictures from a concept list translated to seven languages. We find that this benchmark contains translation errors of varying severity in Spanish, Japanese, and Chinese.
  arXiv  Detail & Related papers  (2024-03-17T05:05:11Z)
• Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation [72.6667341525552]
  We present a new MMT approach based on a strong text-only MT model, which uses neural adapters and a novel guided self-attention mechanism. We also introduce CoMMuTE, a Contrastive Multimodal Translation Evaluation set of ambiguous sentences and their possible translations. Our approach obtains competitive results compared to strong text-only models on standard English-to-French, English-to-German and English-to-Czech benchmarks.
  arXiv  Detail & Related papers  (2022-12-20T10:18:18Z)
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training [52.852163987208826]
  UC2 is the first machine translation-augmented framework for cross-lingual cross-modal representation learning. We propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM) Our proposed framework achieves new state-of-the-art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.
  arXiv  Detail & Related papers  (2021-04-01T08:30:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.