Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

• URL: http://arxiv.org/abs/2006.00843v2
• Date: Tue, 3 Nov 2020 09:26:07 GMT
• Title: Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing
• Authors: Anne Lauscher, Lily Ng, Courtney Napoles, Joel Tetreault
• Abstract summary: We present GAQCorpus: the first large-scale English multi-domain (community Q&A forums, debate forums, review forums) corpus annotated with theory-based AQ scores. We demonstrate the feasibility of large-scale AQ annotation, show that exploiting relations between dimensions yields performance improvements, and explore the synergies between theory-based prediction and practical AQ assessment.
• Score: 6.654552816487819
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Though preceding work in computational argument quality (AQ) mostly focuses on assessing overall AQ, researchers agree that writers would benefit from feedback targeting individual dimensions of argumentation theory. However, a large-scale theory-based corpus and corresponding computational models are missing. We fill this gap by conducting an extensive analysis covering three diverse domains of online argumentative writing and presenting GAQCorpus: the first large-scale English multi-domain (community Q&A forums, debate forums, review forums) corpus annotated with theory-based AQ scores. We then propose the first computational approaches to theory-based assessment, which can serve as strong baselines for future work. We demonstrate the feasibility of large-scale AQ annotation, show that exploiting relations between dimensions yields performance improvements, and explore the synergies between theory-based prediction and practical AQ assessment.

Related papers

• The Foundations of Tokenization: Statistical and Computational Concerns [51.370165245628975]
  Tokenization is a critical step in the NLP pipeline. Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood. The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models.
  arXiv  Detail & Related papers  (2024-07-16T11:12:28Z)
• eRST: A Signaled Graph Theory of Discourse Relations and Organization [14.074017875514787]
  We present a new theoretical framework for computational discourse analysis, based on an expansion of Rhetorical Structure Theory (RST) The framework encompasses discourse relation graphs with tree-breaking, non-projective and concurrent relations, as well as implicit and explicit signals which give explainable rationales to our analyses. We present and evaluate a freely available corpus of English annotated according to our framework, encompassing 12 spoken and written genres with over 200K tokens.
  arXiv  Detail & Related papers  (2024-03-20T12:52:38Z)
• Reinforcement learning for question answering in programming domain using public community scoring as a human feedback [0.0]
  We investigate the enhancement of the GPT Neo 125M performance in Community Question Answering (CQA) with a focus on programming. Two distinct reward model training strategies are employed for fine-tuning with Proximal Policy Optimization (PPO) An auxiliary scoring mechanism is introduced, which demonstrates the limitations of conventional linguistic metrics in evaluating responses in the programming domain.
  arXiv  Detail & Related papers  (2024-01-19T18:49:36Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Investigating the Role of Centering Theory in the Context of Neural Coreference Resolution Systems [71.57556446474486]
  We investigate the connection between centering theory and modern coreference resolution systems. We show that high-quality neural coreference resolvers may not benefit much from explicitly modeling centering ideas. We formulate a version of CT that also models recency and show that it captures coreference information better compared to vanilla CT.
  arXiv  Detail & Related papers  (2022-10-26T12:55:26Z)
• Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review [52.359007622096684]
  Peer review is a key component of the publishing process in most fields of science. Existing NLP studies focus on the analysis of individual texts. editorial assistance often requires modeling interactions between pairs of texts.
  arXiv  Detail & Related papers  (2022-04-22T16:39:38Z)
• Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale [12.883536911500062]
  We study claim quality assessment irrespective of discussed aspects by comparing different revisions of the same claim. We propose two tasks: assessing which claim of a revision pair is better, and ranking all versions of a claim by quality.
  arXiv  Detail & Related papers  (2021-01-25T17:32:04Z)
• Creating a Domain-diverse Corpus for Theory-based Argument Quality Assessment [6.654552816487819]
  We describe GAQCorpus, the first large, domain-diverse annotated corpus of theory-based AQ. We discuss how we designed the annotation task to reliably collect a large number of judgments with crowdsourcing. Our work will inform research on theory-based argumentation annotation and enable the creation of more diverse corpora to support computational AQ assessment.
  arXiv  Detail & Related papers  (2020-11-03T09:40:25Z)
• A Generalised Approach for Encoding and Reasoning with Qualitative Theories in Answer Set Programming [3.963609604649393]
  A family of ASP encodings is proposed which can handle any qualitative calculus with binary relations. This paper is under consideration for acceptance in TPLP.
  arXiv  Detail & Related papers  (2020-08-04T13:31:25Z)
• Invariant Rationalization [84.1861516092232]
  A typical rationalization criterion, i.e. maximum mutual information (MMI), finds the rationale that maximizes the prediction performance based only on the rationale. We introduce a game-theoretic invariant rationalization criterion where the rationales are constrained to enable the same predictor to be optimal across different environments. We show both theoretically and empirically that the proposed rationales can rule out spurious correlations, generalize better to different test scenarios, and align better with human judgments.
  arXiv  Detail & Related papers  (2020-03-22T00:50:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.