A Picture is Worth a Thousand (Correct) Captions: A Vision-Guided Judge-Corrector System for Multimodal Machine Translation

• URL: http://arxiv.org/abs/2511.07010v1
• Date: Mon, 10 Nov 2025 12:02:48 GMT
• Title: A Picture is Worth a Thousand (Correct) Captions: A Vision-Guided Judge-Corrector System for Multimodal Machine Translation
• Abstract summary: We present a two-stage approach that addresses quality issues in the training data through automated error detection and correction.<n>Our methodology introduces a vision-augmented judge-corrector pipeline that leverages multimodal language models.<n>This pipeline processes 28,928 training examples across four languages, correcting an average of 17.1% of captions per language.
• Score: 0.02916558661202723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we describe our system under the team name BLEU Monday for the English-to-Indic Multimodal Translation Task at WAT 2025. We participate in the text-only translation tasks for English-Hindi, English-Bengali, English-Malayalam, and English-Odia language pairs. We present a two-stage approach that addresses quality issues in the training data through automated error detection and correction, followed by parameter-efficient model fine-tuning. Our methodology introduces a vision-augmented judge-corrector pipeline that leverages multimodal language models to systematically identify and correct translation errors in the training data. The judge component classifies translations into three categories: correct, visually ambiguous (requiring image context), or mistranslated (poor translation quality). Identified errors are routed to specialized correctors: GPT-4o-mini regenerates captions requiring visual disambiguation, while IndicTrans2 retranslates cases with pure translation quality issues. This automated pipeline processes 28,928 training examples across four languages, correcting an average of 17.1% of captions per language. We then apply Low-Rank Adaptation (LoRA) to fine-tune the IndicTrans2 en-indic 200M distilled model on both original and corrected datasets. Training on corrected data yields consistent improvements, with BLEU score gains of +1.30 for English-Bengali on the evaluation set (42.00 -> 43.30) and +0.70 on the challenge set (44.90 -> 45.60), +0.60 for English-Odia on the evaluation set (41.00 -> 41.60), and +0.10 for English-Hindi on the challenge set (53.90 -> 54.00).

Related papers

• Generating Adversarial Texts for Machine Translation via GRPO [9.016860740260679]
  We propose a scalable reinforcement-learning-based approach for rewriting existing source texts.<n>On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability.<n> Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data.
  arXiv  Detail & Related papers  (2026-09-05T12:06:59Z)
• Statistical Machine Translation Systems of English-Pnar Language Pair : Some Insights of the Emperical Study [7.0625416160480405]
  Pnar, an Austroasiatic language spoken by approximately 0.4 million people in the Jaintia Hills of Meghalaya, lacks the digital corpora and natural language processing (NLP) resources.<n>This paper presents the first machine translation study for the English and Pnar language pair.
  arXiv  Detail & Related papers  (2026-08-24T11:29:53Z)
• TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model [0.0]
  We present TRACE-BN, a curriculum-guided dataset of structured tutoring traces for Bangla-speaking learners of English.<n>Traces are generated by Gemini 3.5 Flash Lite as the teacher model from NCTB Classes 9-10 English curriculum units.<n>Results show that curriculum-guided structured supervision can transfer multi-component tutoring behavior to a sub-1B model.
  arXiv  Detail & Related papers  (2026-08-15T13:19:17Z)
• The Saturation Point of Backtranslation in High Quality Low Resource English Gujarati Machine Translation [0.0]
  Backtranslation BT is widely used in low resource machine translation MT to generate additional synthetic training data using monolingual corpora.<n>We explore the effectiveness of backtranslation for English Gujarati translation using the multilingual pretrained MBART50 model.
  arXiv  Detail & Related papers  (2025-06-12T09:02:53Z)
• NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts [57.53692236201343]
  We propose a Multi-Task Correction MoE, where we train the experts to become an expert'' of speech-to-text, language-to-text and vision-to-text datasets. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
  arXiv  Detail & Related papers  (2024-11-08T20:11:24Z)
• A Data Selection Approach for Enhancing Low Resource Machine Translation Using Cross-Lingual Sentence Representations [0.4499833362998489]
  This study focuses on the case of English-Marathi language pairs, where existing datasets are notably noisy. To mitigate the impact of data quality issues, we propose a data filtering approach based on cross-lingual sentence representations. Results demonstrate a significant improvement in translation quality over the baseline post-filtering with IndicSBERT.
  arXiv  Detail & Related papers  (2024-09-04T13:49:45Z)
• Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis [3.16714407449467]
  We investigate the role of translation and synthetic data in training language models. We translate TinyStories, a dataset of 2.2M short stories for 3-4 year old children, from English to Arabic using the open NLLB-3B MT model. To rectify these issues, we pre-train the models with a small dataset of synthesized high-quality Arabic stories.
  arXiv  Detail & Related papers  (2024-05-23T07:53:04Z)
• KIT's Multilingual Speech Translation System for IWSLT 2023 [58.5152569458259]
  We describe our speech translation system for the multilingual track of IWSLT 2023. The task requires translation into 10 languages of varying amounts of resources. Our cascaded speech system substantially outperforms its end-to-end counterpart on scientific talk translation.
  arXiv  Detail & Related papers  (2023-06-08T16:13:20Z)
• Large language models effectively leverage document-level context for literary translation, but critical errors persist [32.54546652197316]
  Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. We show through a rigorous human evaluation that asking the Gpt-3.5 (text-davinci-003) LLM to translate an entire literary paragraph results in higher-quality translations.
  arXiv  Detail & Related papers  (2023-04-06T17:27:45Z)
• Easy Guided Decoding in Providing Suggestions for Interactive Machine Translation [14.615314828955288]
  We propose a novel constrained decoding algorithm, namely Prefix Suffix Guided Decoding (PSGD) PSGD improves translation quality by an average of $10.87$ BLEU and $8.62$ BLEU on the WeTS and the WMT 2022 Translation Suggestion datasets.
  arXiv  Detail & Related papers  (2022-11-14T03:40:02Z)
• The Effect of Normalization for Bi-directional Amharic-English Neural Machine Translation [53.907805815477126]
  This paper presents the first relatively large-scale Amharic-English parallel sentence dataset. We build bi-directional Amharic-English translation models by fine-tuning the existing Facebook M2M100 pre-trained model. The results show that the normalization of Amharic homophone characters increases the performance of Amharic-English machine translation in both directions.
  arXiv  Detail & Related papers  (2022-10-27T07:18:53Z)
• Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT22 Translation Task [49.916963624249355]
  This paper describes Tencent AI Lab - Shanghai Jiao Tong University (TAL-SJTU) Low-Resource Translation systems for the WMT22 shared task. We participate in the general translation task on English$Leftrightarrow$Livonian. Our system is based on M2M100 with novel techniques that adapt it to the target language pair.
  arXiv  Detail & Related papers  (2022-10-17T04:34:09Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.