Evaluation Discrepancy Discovery: A Sentence Compression Case-study

• URL: http://arxiv.org/abs/2101.09079v1
• Date: Fri, 22 Jan 2021 12:28:41 GMT
• Title: Evaluation Discrepancy Discovery: A Sentence Compression Case-study
• Abstract summary: We show that neither metric nor human evaluation is sufficient to draw conclusions about system performance. We demonstrate how a system can game a well-established dataset to achieve state-of-the-art results.
• Score: 0.6430130814523793
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Reliable evaluation protocols are of utmost importance for reproducible NLP research. In this work, we show that sometimes neither metric nor conventional human evaluation is sufficient to draw conclusions about system performance. Using sentence compression as an example task, we demonstrate how a system can game a well-established dataset to achieve state-of-the-art results. In contrast with the results reported in previous work that showed correlation between human judgements and metric scores, our manual analysis of state-of-the-art system outputs demonstrates that high metric scores may only indicate a better fit to the data, but not better outputs, as perceived by humans.

Related papers

• Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions [22.299021697499196]
  We propose a classification of assumptions into two categories: task, which can be tested from conversation data alone, and outcome, which requires outcome data and behavioral studies for testing.<n>We retrospectively analyze a healthcare RCT as a case study and find that the gap naturally separates into task and outcome gaps of roughly equal size.
  arXiv  Detail & Related papers  (2026-05-21T15:27:58Z)
• DEEP: Docker-based Execution and Evaluation Platform [0.8666275811953877]
  Our proposed software (DEEP) automates the execution and scoring of machine translation and optical character recognition models.<n>DEEP is prepared to receive dockerized systems, run them (extracting information at that same time), and assess hypothesis against some references.<n>It uses a clustering algorithm based on a statistical analysis of the significance of the results yielded by each model.
  arXiv  Detail & Related papers  (2026-02-23T08:08:57Z)
• On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective [0.0]
  We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split.<n>We demonstrate that the evaluation does not measure generalization to unseen individuals.
  arXiv  Detail & Related papers  (2026-02-19T21:37:42Z)
• Where is this coming from? Making groundedness count in the evaluation of Document VQA models [12.951716701565019]
  We argue that common evaluation metrics do not account for the semantic and multimodal groundedness of a model's outputs. We propose a new evaluation methodology that accounts for the groundedness of predictions. Our proposed methodology is parameterized in such a way that users can configure the score according to their preferences.
  arXiv  Detail & Related papers  (2025-03-24T20:14:46Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• Cobra Effect in Reference-Free Image Captioning Metrics [58.438648377314436]
  A proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. In this paper, we study if there are any deficiencies in reference-free metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-02-18T12:36:23Z)
• Temporal Output Discrepancy for Loss Estimation-based Active Learning [65.93767110342502]
  We present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss. Our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2022-12-20T19:29:37Z)
• Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences [38.919090721583075]
  We show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. We design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Our approach improves faithfulness scores by 20% over strong baselines on AnswerSumm citepfabbri 2021answersumm dataset.
  arXiv  Detail & Related papers  (2022-12-19T18:51:06Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators [91.3755431537592]
  We show that even in arguably the simplest setting, estimation under ignorability assumptions can be misleading. We consider two popular machine learning benchmark datasets for evaluation of heterogeneous treatment effect estimators. We highlight that the inherent characteristics of the benchmark datasets favor some algorithms over others.
  arXiv  Detail & Related papers  (2021-07-28T13:21:27Z)
• Evaluation of Unsupervised Entity and Event Salience Estimation [17.74208462902158]
  Salience Estimation aims to predict term importance in documents. Previous studies typically generate pseudo-ground truth for evaluation. In this work, we propose a light yet practical entity and event salience estimation evaluation protocol.
  arXiv  Detail & Related papers  (2021-04-14T15:23:08Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)
• Critical analysis on the reproducibility of visual quality assessment using deep features [6.746400031322727]
  Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature.
  arXiv  Detail & Related papers  (2020-09-10T09:51:18Z)
• Discourse-Aware Unsupervised Summarization of Long Scientific Documents [13.405733237855404]
  We propose an unsupervised graph-based ranking model for extractive summarization of long scientific documents. Our method assumes a two-level hierarchical graph representation of the source document, and exploits asymmetrical positional cues to determine sentence importance. Results on the PubMed and arXiv datasets show that our approach outperforms strong unsupervised baselines by wide margins in automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-05-01T17:31:11Z)
• Overly Optimistic Prediction Results on Imbalanced Data: a Case Study of Flaws and Benefits when Applying Over-sampling [13.463035357173045]
  We focus on one specific type of methodological flaw: applying over-sampling before partitioning the data into mutually exclusive training and testing sets. We show how this causes the results to be biased using two artificial datasets and reproduce results of studies in which this flaw was identified.
  arXiv  Detail & Related papers  (2020-01-15T12:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.