JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation

• URL: http://arxiv.org/abs/2601.00223v1
• Date: Thu, 01 Jan 2026 06:09:45 GMT
• Title: JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation
• Abstract summary: JP-TL-Bench is a benchmark designed to guide the iterative development of Japanese-English translation systems.<n>It evaluates a candidate model via reference-free, pairwise comparisons against a fixed, versioned anchor set.<n>Because each candidate is scored against the same frozen anchor set, scores are structurally stable given the same base set, judge, and aggregation code.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce JP-TL-Bench, a lightweight, open benchmark designed to guide the iterative development of Japanese-English translation systems. In this context, the challenge is often "which of these two good translations is better?" rather than "is this translation acceptable?" This distinction matters for Japanese-English, where subtle choices in politeness, implicature, ellipsis, and register strongly affect perceived naturalness. JP-TL-Bench uses a protocol built to make LLM judging both reliable and affordable: it evaluates a candidate model via reference-free, pairwise LLM comparisons against a fixed, versioned anchor set. Pairwise results are aggregated with a Bradley-Terry model and reported as win rates plus a normalized 0-10 "LT" score derived from a logistic transform of fitted log-strengths. Because each candidate is scored against the same frozen anchor set, scores are structurally stable given the same base set, judge, and aggregation code.

Related papers

• Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge [0.0]
  We propose a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-switched variants.<n>A reliable judge should preserve its preference under label-accessible language transformations.<n>Across models, Chinese and language-switched presentations induce 10.7--14.4% preference flips relative to English, and all judges achieve their highest accuracy in English.
  arXiv  Detail & Related papers  (2026-06-12T08:59:26Z)
• A Komi-Yazva--Russian Parallel Corpus and Evaluation Protocol for Zero- and Few-Shot LLM Translation [0.0]
  We present the first Komi-Yazva--Russian parallel corpus together with an explicit evaluation protocol.<n>The dataset contains 457 aligned sentence pairs from 74 narrative texts.<n>We use this setup to compare modern large language models on Komi-Yazva-to-Russian translation under severe parallel-data scarcity.
  arXiv  Detail & Related papers  (2026-06-04T17:26:38Z)
• Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets [2.0199251985015434]
  We present a fully automated framework designed to enable scalable, high-quality translation of datasets and benchmarks.<n>We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages.
  arXiv  Detail & Related papers  (2026-02-25T18:58:25Z)
• PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning [54.19784655270799]
  We introduce textbfPEGRL, a textittwo-stage RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization.<n>Experiments on English$to$Finnish, English$to$Turkish, and English$leftrightarrow$Chinese show consistent gains over RL baselines.
  arXiv  Detail & Related papers  (2026-02-03T10:22:55Z)
• NL in the Middle: Code Translation with LLMs and Intermediate Representations [56.77064674776534]
  Large language models (LLMs) produce buggy code translations.<n>One promising avenue to improve translation accuracy is through intermediate representations.<n>We investigate whether LLM-based code translation can benefit from intermediate representations.
  arXiv  Detail & Related papers  (2025-07-11T14:29:21Z)
• LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark [1.3927943269211591]
  We propose a comprehensive framework that enhances Large Language Models (LLMs)-based machine translation evaluation.<n>We extend the ONUBAD dataset by incorporating Sylheti-English sentence pairs, corresponding machine translations, and Direct Assessment (DA) scores annotated by native speakers.<n>Our evaluation shows that the proposed pipeline consistently outperforms existing methods, achieving the highest gain of +0.1083 in Spearman correlation.
  arXiv  Detail & Related papers  (2025-05-18T07:24:13Z)
• Trans-Zero: Self-Play Incentivizes Large Language Models for Multilingual Translation Without Parallel Data [64.4458540273004]
  We propose a self-play framework that leverages only monolingual data and the intrinsic multilingual knowledge of Large Language Models (LLMs)<n>Experiments demonstrate that this approach not only matches the performance of models trained on large-scale parallel data but also excels in non-English translation directions.
  arXiv  Detail & Related papers  (2025-04-20T16:20:30Z)
• Lost in Literalism: How Supervised Training Shapes Translationese in LLMs [51.04435855143767]
  Large language models (LLMs) have achieved remarkable success in machine translation.<n>However, translationese, characterized by overly literal and unnatural translations, remains a persistent challenge.<n>We introduce methods to mitigate these biases, including polishing golden references and filtering unnatural training instances.
  arXiv  Detail & Related papers  (2025-03-06T12:14:45Z)
• Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation [5.689194193929357]
  Large language models (LLMs) have shown superior capabilities in translating figurative language compared to neural machine translation (NMT) systems.<n>This paper introduces two parallel datasets of sentences containing idiomatic expressions for Persian$rightarrow$English and English$rightarrow$Persian translations.<n>We evaluate various open- and closed-source LLMs, NMT models, and their combinations.<n>Experiments reveal that Claude-3.5-Sonnet delivers outstanding results in both translation directions.
  arXiv  Detail & Related papers  (2024-12-13T09:29:27Z)
• Language Models and Cycle Consistency for Self-Reflective Machine Translation [1.79487674052027]
  We generate multiple translation candidates from a source language A to a target language B, and subsequently translate these candidates back to the original language A. By evaluating the cycle consistency between the original and back-translated sentences using metrics such as token-level precision and accuracy, we implicitly estimate the translation quality in language B. For each source sentence, we identify the translation candidate with optimal cycle consistency with the original sentence as the final answer.
  arXiv  Detail & Related papers  (2024-11-05T04:01:41Z)
• MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems [43.19298196163617]
  We present a simple efficient technique to combine the best of both worlds.<n>We develop MIRAGE-Bench, a synthetic arena-based RAG benchmark for 18 diverse languages on Wikipedia.
  arXiv  Detail & Related papers  (2024-10-17T16:18:49Z)
• LLM-based Translation Inference with Iterative Bilingual Understanding [52.46978502902928]
  We propose a novel Iterative Bilingual Understanding Translation method based on the cross-lingual capabilities of large language models (LLMs)<n>The cross-lingual capability of LLMs enables the generation of contextual understanding for both the source and target languages separately.<n>The proposed IBUT outperforms several strong comparison methods.
  arXiv  Detail & Related papers  (2024-10-16T13:21:46Z)
• Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning [57.323716555996114]
  Off-target translation remains an unsolved problem, especially for low-resource languages. Recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs.
  arXiv  Detail & Related papers  (2024-03-21T13:47:40Z)
• TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinement [26.26493253161022]
  Large Language Models (LLMs) have achieved impressive results in Machine Translation (MT) We introduce a systematic LLM-based self-refinement translation framework, named textbfTEaR.
  arXiv  Detail & Related papers  (2024-02-26T07:58:12Z)
• On the Relation between Syntactic Divergence and Zero-Shot Performance [22.195133438732633]
  We take the transfer of Universal Dependencies (UD) parsing from English to a diverse set of languages and conduct two sets of experiments. We analyze zero-shot performance based on the extent to which English source edges are preserved in translation. In both sets of experiments, our results suggest a strong relation between cross-lingual stability and zero-shot parsing performance.
  arXiv  Detail & Related papers  (2021-10-09T21:09:21Z)
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation [55.02832094101173]
  Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual similarity. This paper concerns ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations.
  arXiv  Detail & Related papers  (2020-05-03T22:10:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.