CLEME2.0: Towards More Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction

• URL: http://arxiv.org/abs/2407.00934v1
• Date: Mon, 1 Jul 2024 03:35:58 GMT
• Title: CLEME2.0: Towards More Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction
• Authors: Jingheng Ye, Zishan Xu, Yinghui Li, Xuxin Cheng, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Xin Su,
• Abstract summary: The paper focuses on improving the interpretability of Grammatical Error Correction (GEC) metrics. We propose CLEME2.0, a reference-based evaluation strategy that can describe four elementary dimensions of GEC systems.
• Score: 28.533044857379647
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paper focuses on improving the interpretability of Grammatical Error Correction (GEC) metrics, which receives little attention in previous studies. To bridge the gap, we propose CLEME2.0, a reference-based evaluation strategy that can describe four elementary dimensions of GEC systems, namely hit-correction, error-correction, under-correction, and over-correction. They collectively contribute to revealing the critical characteristics and locating drawbacks of GEC systems. Evaluating systems by Combining these dimensions leads to high human consistency over other reference-based and reference-less metrics. Extensive experiments on 2 human judgement datasets and 6 reference datasets demonstrate the effectiveness and robustness of our method. All the codes will be released after the peer review.

Related papers

• Revisiting Meta-evaluation for Grammatical Error Correction [14.822205658480813]
  SEEDA is a new dataset for GEC meta-evaluation. It consists of corrections with human ratings along two different granularities. The results suggest that edit-based metrics may have been underestimated in existing studies.
  arXiv  Detail & Related papers  (2024-03-05T05:53:09Z)
• Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks [60.82579717007963]
  We introduce an enhanced evaluation framework designed to more accurately gauge the effectiveness, consistency, and overall capability of Graph Contrastive Learning (GCL) methods.
  arXiv  Detail & Related papers  (2024-02-24T01:47:56Z)
• CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation [87.44350003888646]
  Eval-Instruct can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting. CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines.
  arXiv  Detail & Related papers  (2023-11-30T16:52:42Z)
• Grammatical Error Correction via Mixed-Grained Weighted Training [68.94921674855621]
  Grammatical Error Correction (GEC) aims to automatically correct grammatical errors in natural texts. MainGEC designs token-level and sentence-level training weights based on inherent discrepancies in accuracy and potential diversity of data annotation.
  arXiv  Detail & Related papers  (2023-11-23T08:34:37Z)
• RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation [64.2568239429946]
  We introduce RobustGEC, a benchmark designed to evaluate the context robustness of GEC systems. We reveal that state-of-the-art GEC systems still lack sufficient robustness against context perturbations.
  arXiv  Detail & Related papers  (2023-10-11T08:33:23Z)
• Evaluation of really good grammatical error correction [0.0]
  Grammatical Error Correction (GEC) encompasses various models with distinct objectives. Traditional evaluation methods fail to fully capture the full range of system capabilities and objectives.
  arXiv  Detail & Related papers  (2023-08-17T13:45:35Z)
• CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction [32.44051877804761]
  Chunk-LEvel Multi-reference Evaluation (CLEME) is designed to evaluate Grammatical Error Correction (GEC) systems in the multi-reference evaluation setting. We conduct experiments on six English reference sets based on the CoNLL-2014 shared task.
  arXiv  Detail & Related papers  (2023-05-18T08:57:17Z)
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation [136.16507050034755]
  Existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale. We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units. We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems.
  arXiv  Detail & Related papers  (2022-12-15T17:26:05Z)
• A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction [54.569707226277735]
  Existing approaches for grammatical error correction (GEC) rely on supervised learning with manually created GEC datasets. There is a non-negligible amount of "noise" where errors were inappropriately edited or left uncorrected. We propose a self-refinement method where the key idea is to denoise these datasets by leveraging the prediction consistency of existing models.
  arXiv  Detail & Related papers  (2020-10-07T04:45:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.