Minimum Bayes' Risk Decoding for System Combination of Grammatical Error
Correction Systems
- URL: http://arxiv.org/abs/2309.06520v2
- Date: Fri, 27 Oct 2023 14:42:29 GMT
- Title: Minimum Bayes' Risk Decoding for System Combination of Grammatical Error
Correction Systems
- Authors: Vyas Raina and Mark Gales
- Abstract summary: This paper examines Minimum Bayes' Risk (MBR) decoding for Grammatical Error Correction (GEC) systems.
We propose a novel MBR loss function directly linked to this form of criterion.
Experiments on three popular GEC datasets and with state-of-the-art GEC systems demonstrate the efficacy of the proposed approach.
- Score: 3.722707313671672
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: For sequence-to-sequence tasks it is challenging to combine individual system
outputs. Further, there is also often a mismatch between the decoding criterion
and the one used for assessment. Minimum Bayes' Risk (MBR) decoding can be used
to combine system outputs in a manner that encourages better alignment with the
final assessment criterion. This paper examines MBR decoding for Grammatical
Error Correction (GEC) systems, where performance is usually evaluated in terms
of edits and an associated F-score. Hence, we propose a novel MBR loss function
directly linked to this form of criterion. Furthermore, an approach to expand
the possible set of candidate sentences is described. This builds on a current
max-voting combination scheme, as well as individual edit-level selection.
Experiments on three popular GEC datasets and with state-of-the-art GEC systems
demonstrate the efficacy of the proposed MBR approach. Additionally, the paper
highlights how varying reward metrics within the MBR decoding framework can
provide control over precision, recall, and the F-score in combined GEC
systems.
Related papers
- Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms [19.543681023903456]
We formulate Minimum Bayes Risk (MBR) decoding as a matrix completion problem.
We exploit this by only computing a random subset of the scores and efficiently recover the missing entries in the matrix.
Our experimental results on machine translation tasks demonstrate that the proposed method requires 1/16 utility metric computations.
arXiv Detail & Related papers (2024-06-05T00:54:03Z) - LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction [49.0746090186582]
Over-correction is a critical problem in Chinese grammatical error correction (CGEC) task.
Recent work using model ensemble methods can effectively mitigate over-correction and improve the precision of the GEC system.
We propose the LM-Combiner, a rewriting model that can directly modify the over-correction of GEC system outputs without a model ensemble.
arXiv Detail & Related papers (2024-03-26T06:12:21Z) - It's MBR All the Way Down: Modern Generation Techniques Through the Lens
of Minimum Bayes Risk [57.641436861482696]
Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lowest risk (expected error) among multiple candidates.
arXiv Detail & Related papers (2023-10-02T17:47:10Z) - CLEME: Debiasing Multi-reference Evaluation for Grammatical Error
Correction [32.44051877804761]
Chunk-LEvel Multi-reference Evaluation (CLEME) is designed to evaluate Grammatical Error Correction (GEC) systems in the multi-reference evaluation setting.
We conduct experiments on six English reference sets based on the CoNLL-2014 shared task.
arXiv Detail & Related papers (2023-05-18T08:57:17Z) - DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding [53.33313271531839]
Minimum Bayesian Risk Decoding (MBR) emerges as a promising decoding algorithm in Neural Machine Translation.
MBR performs poorly with label smoothing, which is surprising as label smoothing provides decent improvement with beam search and improves generality in various tasks.
We show that the issue arises from the un-consistency of label smoothing on the token-level and sequence-level distributions.
arXiv Detail & Related papers (2022-12-08T11:40:31Z) - Gender Bias and Universal Substitution Adversarial Attacks on
Grammatical Error Correction Systems for Automated Assessment [1.4213973379473654]
GEC systems are often used on speech transcriptions of English learners as a form of assessment and feedback.
The count of edits from a candidate's input sentence to a GEC system's grammatically corrected output sentence is indicative of a candidate's language ability.
This work examines a simple universal substitution adversarial attack that non-native speakers of English could realistically employ to deceive GEC systems used for assessment.
arXiv Detail & Related papers (2022-08-19T17:44:13Z) - Large-Scale Sequential Learning for Recommender and Engineering Systems [91.3755431537592]
In this thesis, we focus on the design of an automatic algorithms that provide personalized ranking by adapting to the current conditions.
For the former, we propose novel algorithm called SAROS that take into account both kinds of feedback for learning over the sequence of interactions.
The proposed idea of taking into account the neighbour lines shows statistically significant results in comparison with the initial approach for faults detection in power grid.
arXiv Detail & Related papers (2022-05-13T21:09:41Z) - Integrate Lattice-Free MMI into End-to-End Speech Recognition [87.01137882072322]
In automatic speech recognition (ASR) research, discriminative criteria have achieved superior performance in DNN-HMM systems.
With this motivation, the adoption of discriminative criteria is promising to boost the performance of end-to-end (E2E) ASR systems.
Previous works have introduced the minimum Bayesian risk (MBR, one of the discriminative criteria) into E2E ASR systems.
In this work, novel algorithms are proposed in this work to integrate another widely used discriminative criterion, lattice-free maximum mutual information (LF-MMI) into E2E
arXiv Detail & Related papers (2022-03-29T14:32:46Z) - Sequence Transduction with Graph-based Supervision [96.04967815520193]
We present a new transducer objective function that generalizes the RNN-T loss to accept a graph representation of the labels.
We demonstrate that transducer-based ASR with CTC-like lattice achieves better results compared to standard RNN-T.
arXiv Detail & Related papers (2021-11-01T21:51:42Z) - Diversity-Driven Combination for Grammatical Error Correction [30.63256303821261]
Grammatical error correction (GEC) is the task of detecting and correcting errors in a written text.
To achieve successful system combination, multiple component systems need to produce corrected sentences that are both diverse and of comparable quality.
We present Diversity-Driven Combination (DDC) for GEC, a system combination strategy that encourages diversity among component systems.
arXiv Detail & Related papers (2021-10-28T14:20:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.