Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

• URL: http://arxiv.org/abs/2309.11925v1
• Date: Thu, 21 Sep 2023 09:38:56 GMT
• Title: Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task
• Authors: Ricardo Rei, Nuno M. Guerreiro, Jos\'e Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, Jos\'e G.C. de Souza, Andr\'e F.T. Martins
• Abstract summary: We present the joint contribution of Unbabel and Instituto Superior T'ecnico to the WMT 2023 Shared Task on Quality Estimation (QE) Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2) Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level judgements.
• Score: 11.681598828340912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the joint contribution of Unbabel and Instituto Superior T\'ecnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2). For all tasks, we build on the COMETKIWI-22 model (Rei et al., 2022b). Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level granularity. Compared to the previous state-of-the-art COMETKIWI-22, we show large improvements in correlation with human judgements (up to 10 Spearman points). Moreover, we surpass the second-best multilingual submission to the shared-task with up to 3.8 absolute points.

Related papers

• Rethinking and Improving Multi-task Learning for End-to-end Speech Translation [51.713683037303035]
  We investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. We propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation.
  arXiv  Detail & Related papers  (2023-11-07T08:48:46Z)
• Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task [59.46906545506715]
  We introduce the NJUNLP team to the WMT 2023 Quality Estimation (QE) shared task. Our team submitted predictions for the English-German language pair on all two sub-tasks. Our models achieved the best results in English-German for both word-level and fine-grained error span detection sub-tasks.
  arXiv  Detail & Related papers  (2023-09-23T01:52:14Z)
• NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track [78.80683163990446]
  This paper presents NAVER LABS Europe's systems for Tamasheq-French and Quechua-Spanish speech translation in the IWSLT 2023 Low-Resource track. Our work attempts to maximize translation quality in low-resource settings using multilingual parameter-efficient solutions.
  arXiv  Detail & Related papers  (2023-06-13T13:22:30Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5 [50.574918785575655]
  We compare sequential fine-tuning with a model for multi-task learning in the context of boosting performance on two tasks. Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.
  arXiv  Detail & Related papers  (2022-10-31T13:26:08Z)
• CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task [11.716878242203267]
  We present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE) Our team participated on all three subtasks: (i) Sentence and Word-level Quality Prediction; (ii) Explainable QE; and (iii) Critical Error Detection.
  arXiv  Detail & Related papers  (2022-09-13T18:05:12Z)
• UniTE: Unified Translation Evaluation [63.58868113074476]
  UniTE is the first unified framework engaged with abilities to handle all three evaluation tasks. We testify our framework on WMT 2019 Metrics and WMT 2020 Quality Estimation benchmarks.
  arXiv  Detail & Related papers  (2022-04-28T08:35:26Z)
• Handshakes AI Research at CASE 2021 Task 1: Exploring different approaches for multilingual tasks [0.22940141855172036]
  The aim of the CASE 2021 Shared Task 1 was to detect and classify socio-political and crisis event information in a multilingual setting. Our submission contained entries in all of the subtasks, and the scores obtained validated our research finding.
  arXiv  Detail & Related papers  (2021-10-29T07:58:49Z)
• Ensemble Fine-tuned mBERT for Translation Quality Estimation [0.0]
  In this paper, we discuss our submission to the WMT 2021 QE Shared Task. Our proposed system is an ensemble of multilingual BERT (mBERT)-based regression models. It demonstrates comparable performance with respect to the Pearson's correlation and beats the baseline system in MAE/ RMSE for several language pairs.
  arXiv  Detail & Related papers  (2021-09-08T20:13:06Z)
• Zhestyatsky at SemEval-2021 Task 2: ReLU over Cosine Similarity for BERT Fine-tuning [0.07614628596146598]
  This paper presents our contribution to SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC) Our experiments cover English (EN-EN) sub-track from the multilingual setting of the task. We find the combination of Cosine Similarity and ReLU activation leading to the most effective fine-tuning procedure.
  arXiv  Detail & Related papers  (2021-04-13T18:28:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.