Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting

• URL: http://arxiv.org/abs/2305.14755v2
• Date: Mon, 23 Oct 2023 17:11:30 GMT
• Title: Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting
• Authors: Akhila Yerukola, Xuhui Zhou, Elizabeth Clark, Maarten Sap
• Abstract summary: We introduce a new composite contextual evaluation metric $textttCtxSimFit$ that combines similarity to the original sentence with contextual cohesiveness. Our experiments show that humans significantly prefer contextual rewrites as more fitting and natural over non-contextual ones.
• Score: 29.983234538677543
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambiguous, and incoherent rewrites. In this paper, we investigate integrating the preceding textual context into both the $\textit{rewriting}$ and $\textit{evaluation}$ stages of stylistic text rewriting, and introduce a new composite contextual evaluation metric $\texttt{CtxSimFit}$ that combines similarity to the original sentence with contextual cohesiveness. We comparatively evaluate non-contextual and contextual rewrites in formality, toxicity, and sentiment transfer tasks. Our experiments show that humans significantly prefer contextual rewrites as more fitting and natural over non-contextual ones, yet existing sentence-level automatic metrics (e.g., ROUGE, SBERT) correlate poorly with human preferences ($\rho$=0--0.3). In contrast, human preferences are much better reflected by both our novel $\texttt{CtxSimFit}$ ($\rho$=0.7--0.9) as well as proposed context-infused versions of common metrics ($\rho$=0.4--0.7). Overall, our findings highlight the importance of integrating context into the generation and especially the evaluation stages of stylistic text rewriting.

Related papers

• Fine-grained Controllable Text Generation through In-context Learning with Feedback [57.396980277089135]
  We present a method for rewriting an input sentence to match specific values of nontrivial linguistic features, such as dependency depth. In contrast to earlier work, our method uses in-context learning rather than finetuning, making it applicable in use cases where data is sparse.
  arXiv  Detail & Related papers  (2024-06-17T08:55:48Z)
• CoheSentia: A Novel Benchmark of Incremental versus Holistic Assessment of Coherence in Generated Texts [15.866519123942457]
  We introduce sc CoheSentia, a novel benchmark of human-perceived coherence of automatically generated texts. Our benchmark contains 500 automatically-generated and human-annotated paragraphs, each annotated in both methods. Our analysis shows that the inter-annotator agreement in the incremental mode is higher than in the holistic alternative.
  arXiv  Detail & Related papers  (2023-10-25T03:21:20Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• A General Contextualized Rewriting Framework for Text Summarization [15.311467109946571]
  Exiting rewriting systems take each extractive sentence as the only input, which is relatively focused but can lose necessary background knowledge and discourse context. We formalize contextualized rewriting as a seq2seq with group-tag alignments, identifying extractive sentences through content-based addressing. Results show that our approach significantly outperforms non-contextualized rewriting systems without requiring reinforcement learning.
  arXiv  Detail & Related papers  (2022-07-13T03:55:57Z)
• CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning [65.57338873921168]
  Localizing text instances in natural scenes is regarded as a fundamental challenge in computer vision. In this work, we quantitatively analyze the sub-text problem and present a simple yet effective design, COntrastive RElation (CORE) module. We integrate the CORE module into a two-stage text detector of Mask R-CNN and devise our text detector CORE-Text.
  arXiv  Detail & Related papers  (2021-12-14T16:22:25Z)
• Video Text Tracking With a Spatio-Temporal Complementary Model [46.99051486905713]
  Text tracking is to track multiple texts in a video,and construct a trajectory for each text. Existing methodle this task by utilizing the tracking-by-detection frame-work. We argue that the tracking accuracy of this paradigmis severely limited in more complex scenarios.
  arXiv  Detail & Related papers  (2021-11-09T08:23:06Z)
• Sentence Semantic Regression for Text Generation [25.16392119801612]
  We propose a new framework named Sentence Semantic Regression (textbfSSR) based on sentence-level language modeling. For idea reasoning, two architectures textbfSSR-AR and textbfSSR-NonAR are designed to conduct sentence semantic regression autoregressively. In the phase of surface realization, a mixed-granularity sentence decoder is designed to generate text with better consistency.
  arXiv  Detail & Related papers  (2021-08-06T07:35:59Z)
• A Case Study of Spanish Text Transformations for Twitter Sentiment Analysis [1.9694608733361543]
  Sentiment analysis is a text mining task that determines the polarity of a given text, i.e., its positiveness or negativeness. New forms of textual expressions present new challenges to analyze text given the use of slang, orthographic and grammatical errors.
  arXiv  Detail & Related papers  (2021-06-03T17:24:31Z)
• Contextual Text Style Transfer [73.66285813595616]
  Contextual Text Style Transfer aims to translate a sentence into a desired style with its surrounding context taken into account. We propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context. Two new benchmarks, Enron-Context and Reddit-Context, are introduced for formality and offensiveness style transfer.
  arXiv  Detail & Related papers  (2020-04-30T23:01:12Z)
• Extractive Summarization as Text Matching [123.09816729675838]
  This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. We formulate the extractive summarization task as a semantic text matching problem. We have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1)
  arXiv  Detail & Related papers  (2020-04-19T08:27:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.