Evaluative Judgement in Teaching AI-based Translation: A Class-room Case Study of AI-Mediated Translation and Post-Editing

• URL: http://arxiv.org/abs/2606.15483v1
• Date: Sat, 13 Jun 2026 21:34:50 GMT
• Title: Evaluative Judgement in Teaching AI-based Translation: A Class-room Case Study of AI-Mediated Translation and Post-Editing
• Abstract summary: This paper draws on 23 anonymized student pro-jects from a fourth-year Machine Transla-tion and Post-editing course in a BA-level translation programme.<n>Students translat-ed short English Wikipedia texts into Catalan or Spanish, generated four system outputs, evaluated them using automatic metrics and human adequacy/fluency assessment, selected one output for post-editing, and justified their deci-sion in written reports.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Drawing on 23 anonymized student pro-jects from a fourth-year Machine Transla-tion and Post-editing course in a BA-level translation programme, this paper exam-ines how structured comparison of gen-eral-purpose LLMs and online MT sys-tems can elicit evaluative judgement in AI-mediated translation. Students translat-ed short specialised English Wikipedia texts into Catalan or Spanish, generated four system outputs, evaluated them using automatic metrics and human adequa-cy/fluency assessment, selected one output for post-editing, and justified their deci-sion in written reports. Descriptive counts are reported for all 23 projects, while qualitative interpretation is based on the 22 cases accompanied by written reports. Results show that students did not treat automatic metrics as final authority: final post-editing selections often diverged from metric rankings and were justified through adequacy, fluency, terminology, naturalness, and expected post-editing ef-fort. The study therefore does not bench-mark systems under controlled conditions; it analyses how students justified system choice within an authentic classroom as-signment.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• The Role of Prompt Language and Translation-Theory-Driven Prompts in Large Language Models: A Case Study on Spanish-Chinese Journalistic Translation [13.47712128273878]
  This study examines how prompt language and translation theory-driven prompt design influence the quality of Spanish-Chinese journalistic translations.<n>A parallel corpus of four editorials from El Pais was translated under 48 experimental conditions.
  arXiv  Detail & Related papers  (2026-07-03T09:59:36Z)
• A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language [12.151730134044696]
  This paper investigates the impact of subword tokenization algorithms and self-supervised learning models from different linguistic and acoustic perspectives.<n>It shows that these algorithms fail to adequately describe the downstream application of automatic transcripts.
  arXiv  Detail & Related papers  (2026-05-05T12:39:40Z)
• XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics [64.77152900881724]
  We propose XQ-MEval, a semi-automatically built dataset covering nine translation directions.<n>We inject MQM-defined errors into gold translations automatically, filter them by native speakers for reliability, and merge errors to generate pseudo translations with controllable quality.<n>Using XQ-MEval, our experiments on nine representative metrics reveal the inconsistency between averaging and human judgment.
  arXiv  Detail & Related papers  (2026-04-16T12:27:10Z)
• I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations [9.275967682881944]
  This paper introduces a comprehensive benchmark for evaluating how Large Language Models respond to linguistic shibboleths.<n>We demonstrate how LLMs systematically penalize certain linguistic patterns, particularly hedging language, despite equivalent content quality.<n>We validate our approach along multiple linguistic dimensions, showing that hedged responses receive 25.6% lower ratings on average.
  arXiv  Detail & Related papers  (2025-08-06T23:51:03Z)
• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• QE4PE: Word-level Quality Estimation for Human Post-Editing [17.17222014168155]
  We investigate the impact of word-level QE on machine translation post-editing in a realistic setting involving 42 professional post-editors.<n>We compare four errorspan highlight modalities, including supervised and uncertainty-based word-level QE methods, for identifying potential errors in the outputs of a state-of-the-art neural MT model.<n>We find that domain, language and editors' speed are critical factors in determining highlights' effectiveness, with modest differences between human-made and automated QE.
  arXiv  Detail & Related papers  (2025-03-04T22:50:17Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics [64.88815792555451]
  We show that current methods for judging metrics are highly sensitive to the translations used for assessment. We develop a method for thresholding performance improvement under an automatic metric against human judgements.
  arXiv  Detail & Related papers  (2020-06-11T09:12:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.