Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation

• URL: http://arxiv.org/abs/2208.11646v2
• Date: Thu, 25 Aug 2022 12:43:27 GMT
• Title: Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation
• Authors: Cyril Chhun, Pierre Colombo, Chlo\'e Clavel, Fabian M. Suchanek
• Abstract summary: There is no consensus on which human evaluation criteria to use. No analysis of how well automatic criteria correlate with them. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria.
• Score: 9.299255585127158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.

Related papers

• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• Automatic Answerability Evaluation for Question Generation [32.1067137848404]
  This work proposes PMAN, a novel automatic evaluation metric to assess whether the generated questions are answerable by the reference answers. Our implementation of a GPT-based QG model achieves state-of-the-art performance in generating answerable questions.
  arXiv  Detail & Related papers  (2023-09-22T00:13:07Z)
• HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation [18.049566239050762]
  Proper evaluation metrics are like a beacon guiding the research of simile generation (SG) To address the issues, we establish HA, a holistic and automatic evaluation system for the SG task, which consists of five criteria from three perspectives and automatic metrics for each criterion. Our metrics are significantly more correlated with human ratings from each perspective compared with prior automatic metrics.
  arXiv  Detail & Related papers  (2023-06-13T06:06:01Z)
• An Investigation of Evaluation Metrics for Automated Medical Note Generation [5.094623170336122]
  We study evaluation methods and metrics for the automatic generation of clinical notes from medical conversations. To study the correlation between the automatic metrics and manual judgments, we evaluate automatic notes/summaries by comparing the system and reference facts.
  arXiv  Detail & Related papers  (2023-05-27T04:34:58Z)
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation [136.16507050034755]
  Existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale. We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units. We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems.
  arXiv  Detail & Related papers  (2022-12-15T17:26:05Z)
• Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation [58.54483567073125]
  We propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists. We observed good levels of inter-annotator agreement in a first evaluation study using the protocol.
  arXiv  Detail & Related papers  (2022-11-17T10:54:28Z)
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation [60.59732704936083]
  We take a step back and analyze recent progress by comparing the body of existing automatic metrics and human metrics. Our extensive statistical analysis reveals surprising findings: automatic metrics -- old and new -- are much more similar to each other than to humans.
  arXiv  Detail & Related papers  (2022-08-31T01:13:46Z)
• Perturbation CheckLists for Evaluating NLG Evaluation Metrics [16.20764980129339]
  Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG is inadequate.
  arXiv  Detail & Related papers  (2021-09-13T08:26:26Z)
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach [84.02388020258141]
  We propose a new framework named ENIGMA for estimating human evaluation scores based on off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.
  arXiv  Detail & Related papers  (2021-02-20T03:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.