Analysing Zero-Shot Readability-Controlled Sentence Simplification

• URL: http://arxiv.org/abs/2409.20246v1
• Date: Mon, 30 Sep 2024 12:36:25 GMT
• Title: Analysing Zero-Shot Readability-Controlled Sentence Simplification
• Authors: Abdullah Barayan, Jose Camacho-Collados, Fernando Alva-Manchego,
• Abstract summary: We investigate how different types of contextual information affect a model's ability to generate sentences with the desired readability. Results show that all tested models struggle to simplify sentences due to models' limitations and characteristics of the source sentences. Our experiments also highlight the need for better automatic evaluation metrics tailored to RCTS.
• Score: 54.09069745799918
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Readability-controlled text simplification (RCTS) rewrites texts to lower readability levels while preserving their meaning. RCTS models often depend on parallel corpora with readability annotations on both source and target sides. Such datasets are scarce and difficult to curate, especially at the sentence level. To reduce reliance on parallel data, we explore using instruction-tuned large language models for zero-shot RCTS. Through automatic and manual evaluations, we examine: (1) how different types of contextual information affect a model's ability to generate sentences with the desired readability, and (2) the trade-off between achieving target readability and preserving meaning. Results show that all tested models struggle to simplify sentences (especially to the lowest levels) due to models' limitations and characteristics of the source sentences that impede adequate rewriting. Our experiments also highlight the need for better automatic evaluation metrics tailored to RCTS, as standard ones often misinterpret common simplification operations, and inaccurately assess readability and meaning preservation.

Related papers

• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• Factual Dialogue Summarization via Learning from Large Language Models [35.63037083806503]
  Large language model (LLM)-based automatic text summarization models generate more factually consistent summaries. We employ zero-shot learning to extract symbolic knowledge from LLMs, generating factually consistent (positive) and inconsistent (negative) summaries. Our approach achieves better factual consistency while maintaining coherence, fluency, and relevance, as confirmed by various automatic evaluation metrics.
  arXiv  Detail & Related papers  (2024-06-20T20:03:37Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Evaluating Factual Consistency of Texts with Semantic Role Labeling [3.1776833268555134]
  We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. A final factuality score is computed by an adjustable scoring mechanism. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T17:59:42Z)
• Zero-Shot Text Classification via Self-Supervised Tuning [46.9902502503747]
  We propose a new paradigm based on self-supervised learning to solve zero-shot text classification tasks. tuning the language models with unlabeled data, called self-supervised tuning. Our model outperforms the state-of-the-art baselines on 7 out of 10 tasks.
  arXiv  Detail & Related papers  (2023-05-19T05:47:33Z)
• SNaC: Coherence Error Detection for Narrative Summarization [73.48220043216087]
  We introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators.
  arXiv  Detail & Related papers  (2022-05-19T16:01:47Z)
• Evaluating Factuality in Text Simplification [43.94402649899681]
  We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplification datasets and state-of-the-art model outputs. We find that errors often appear in both that are not captured by existing evaluation metrics.
  arXiv  Detail & Related papers  (2022-04-15T17:37:09Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)
• Small but Mighty: New Benchmarks for Split and Rephrase [18.959219419951083]
  Split and Rephrase is a text simplification task of rewriting a complex sentence into simpler ones. We find that the widely used benchmark dataset universally contains easily exploitable syntactic cues. We show that even a simple rule-based model can perform on par with the state-of-the-art model.
  arXiv  Detail & Related papers  (2020-09-17T23:37:33Z)
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning [85.33459673197149]
  We introduce a new Reading dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. In this paper, we propose to identify biased data points and separate them into EASY set and the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.
  arXiv  Detail & Related papers  (2020-02-11T11:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.