Analyzing Neural Discourse Coherence Models
- URL: http://arxiv.org/abs/2011.06306v1
- Date: Thu, 12 Nov 2020 10:44:41 GMT
- Title: Analyzing Neural Discourse Coherence Models
- Authors: Youmna Farag, Josef Valvoda, Helen Yannakoudakis and Ted Briscoe
- Abstract summary: We investigate how well current models of coherence can capture aspects of text implicated in discourse organisation.
We devise two datasets of various linguistic alterations that undermine coherence and test model sensitivity to changes in syntax and semantics.
- Score: 17.894463722947542
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this work, we systematically investigate how well current models of
coherence can capture aspects of text implicated in discourse organisation. We
devise two datasets of various linguistic alterations that undermine coherence
and test model sensitivity to changes in syntax and semantics. We furthermore
probe discourse embedding space and examine the knowledge that is encoded in
representations of coherence. We hope this study shall provide further insight
into how to frame the task and improve models of coherence assessment further.
Finally, we make our datasets publicly available as a resource for researchers
to use to test discourse coherence models.
Related papers
- How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored.
Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges.
We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
arXiv Detail & Related papers (2023-11-14T08:51:00Z) - RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models [57.12888828853409]
RAVEN is a model that combines retrieval-augmented masked language modeling and prefix language modeling.
Fusion-in-Context Learning enables the model to leverage more in-context examples without requiring additional training.
Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning.
arXiv Detail & Related papers (2023-08-15T17:59:18Z) - Feature Interactions Reveal Linguistic Structure in Language Models [2.0178765779788495]
We study feature interactions in the context of feature attribution methods for post-hoc interpretability.
We work out a grey box methodology, in which we train models to perfection on a formal language classification task.
We show that under specific configurations, some methods are indeed able to uncover the grammatical rules acquired by a model.
arXiv Detail & Related papers (2023-06-21T11:24:41Z) - A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
We propose a novel latent-variable formulation for constructing intrinsic probes.
We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
arXiv Detail & Related papers (2022-01-20T15:01:12Z) - Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction
from Language Models [7.5413579967970605]
We design an extendable set of test suites addressing different aspects of discourse and dialogue coherence.
Unlike most previous coherence evaluation studies, we address specific linguistic devices beyond sentence order perturbations.
We show that this paradigm is equally suited to evaluate linguistic qualities that contribute to the notion of coherence.
arXiv Detail & Related papers (2021-05-07T20:28:33Z) - "Let's Eat Grandma": When Punctuation Matters in Sentence Representation
for Sentiment Analysis [13.873803872380229]
We argue that punctuation could play a significant role in sentiment analysis and propose a novel representation model to improve syntactic and contextual performance.
We conduct experiments on publicly available datasets and verify that our model can identify the sentiments more accurately over other state-of-the-art baseline methods.
arXiv Detail & Related papers (2020-12-10T19:07:31Z) - Neuro-Symbolic Representations for Video Captioning: A Case for
Leveraging Inductive Biases for Vision and Language [148.0843278195794]
We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning.
Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
arXiv Detail & Related papers (2020-11-18T20:21:19Z) - Probing Task-Oriented Dialogue Representation from Language Models [106.02947285212132]
This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks.
We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way.
arXiv Detail & Related papers (2020-10-26T21:34:39Z) - Analysing Lexical Semantic Change with Contextualised Word
Representations [7.071298726856781]
We propose a novel method that exploits the BERT neural language model to obtain representations of word usages.
We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements.
arXiv Detail & Related papers (2020-04-29T12:18:14Z) - How Far are We from Effective Context Modeling? An Exploratory Study on
Semantic Parsing in Context [59.13515950353125]
We present a grammar-based decoding semantic parsing and adapt typical context modeling methods on top of it.
We evaluate 13 context modeling methods on two large cross-domain datasets, and our best model achieves state-of-the-art performances.
arXiv Detail & Related papers (2020-02-03T11:28:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.