Dialect-robust Evaluation of Generated Text
- URL: http://arxiv.org/abs/2211.00922v1
- Date: Wed, 2 Nov 2022 07:12:23 GMT
- Title: Dialect-robust Evaluation of Generated Text
- Authors: Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan
Garrette, Aditya Siddhant, Jacob Eisenstein, Sebastian Gehrmann
- Abstract summary: We formalize dialect robustness and dialect awareness as goals for NLG evaluation metrics.
Applying the suite to current state-of-the-art metrics, we demonstrate that they are not dialect-robust.
- Score: 40.85375247260744
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Evaluation metrics that are not robust to dialect variation make it
impossible to tell how well systems perform for many groups of users, and can
even penalize systems for producing text in lower-resource dialects. However,
currently, there exists no way to quantify how metrics respond to change in the
dialect of a generated utterance. We thus formalize dialect robustness and
dialect awareness as goals for NLG evaluation metrics. We introduce a suite of
methods and corresponding statistical tests one can use to assess metrics in
light of the two goals. Applying the suite to current state-of-the-art metrics,
we demonstrate that they are not dialect-robust and that semantic perturbations
frequently lead to smaller decreases in a metric than the introduction of
dialect features. As a first step to overcome this limitation, we propose a
training schema, NANO, which introduces regional and language information to
the pretraining process of a metric. We demonstrate that NANO provides a
size-efficient way for models to improve the dialect robustness while
simultaneously improving their performance on the standard metric benchmark.
Related papers
- Unsupervised Approach to Evaluate Sentence-Level Fluency: Do We Really
Need Reference? [3.2528685897001455]
This paper adapts an existing unsupervised technique for measuring text fluency without the need for any reference.
Our approach leverages various word embeddings and trains language models using Recurrent Neural Network (RNN) architectures.
To assess the performance of the models, we conduct a comparative analysis across 10 Indic languages.
arXiv Detail & Related papers (2023-12-03T20:09:23Z) - A Benchmark for Evaluating Machine Translation Metrics on Dialects
Without Standard Orthography [40.04973667048665]
We evaluate how robust metrics are to non-standardized dialects.
We collect a dataset of human translations and human judgments for automatic machine translations from English to two Swiss German dialects.
arXiv Detail & Related papers (2023-11-28T15:12:11Z) - HyPoradise: An Open Baseline for Generative Speech Recognition with
Large Language Models [81.56455625624041]
We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction.
The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses.
LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
arXiv Detail & Related papers (2023-09-27T14:44:10Z) - Towards dialect-inclusive recognition in a low-resource language: are
balanced corpora the answer? [5.1121440213561335]
This study is a diagnostic to quantify the effect of the speaker's dialect on recognition performance.
12 ASR systems were trained using dialect-balanced training corpora and modified versions of the baseline corpora.
Results indicate that dialect-balanced corpora do not yield a similar performance across the dialects.
There is a close relationship between Co and Mu dialects, but one that is not symmetrical.
arXiv Detail & Related papers (2023-07-14T12:18:38Z) - Multi-VALUE: A Framework for Cross-Dialectal English NLP [49.55176102659081]
Multi- Dialect is a controllable rule-based translation system spanning 50 English dialects.
Stress tests reveal significant performance disparities for leading models on non-standard dialects.
We partner with native speakers of Chicano and Indian English to release new gold-standard variants of the popular CoQA task.
arXiv Detail & Related papers (2022-12-15T18:17:01Z) - SMART: Sentences as Basic Units for Text Evaluation [48.5999587529085]
In this paper, we introduce a new metric called SMART to mitigate such limitations.
We treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft-match candidate and reference sentences.
Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics.
arXiv Detail & Related papers (2022-08-01T17:58:05Z) - When Does Translation Require Context? A Data-driven, Multilingual
Exploration [71.43817945875433]
proper handling of discourse significantly contributes to the quality of machine translation (MT)
Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation.
We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
arXiv Detail & Related papers (2021-09-15T17:29:30Z) - Discrete representations in neural models of spoken language [56.29049879393466]
We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language.
We find that the different evaluation metrics can give inconsistent results.
arXiv Detail & Related papers (2021-05-12T11:02:02Z) - Fast and Robust Unsupervised Contextual Biasing for Speech Recognition [16.557586847398778]
We propose an alternative approach that does not entail explicit contextual language model.
We derive the bias score for every word in the system vocabulary from the training corpus.
We show significant improvement in recognition accuracy when the relevant context is available.
arXiv Detail & Related papers (2020-05-04T17:29:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.