Discovering influential text using convolutional neural networks

• URL: http://arxiv.org/abs/2406.10086v2
• Date: Fri, 21 Jun 2024 18:14:42 GMT
• Title: Discovering influential text using convolutional neural networks
• Authors: Megan Ayers, Luke Sanford, Margaret Roberts, Eddie Yang,
• Abstract summary: We present a method for discovering clusters of similar text phrases that are predictive of human reactions to texts using convolutional neural networks. We apply the method to two datasets. The first enables direct validation of the model's ability to detect phrases known to cause the outcome. In both cases, the model learns a greater variety of text treatments compared to benchmark methods, and these text features quantitatively meet or exceed the ability of benchmark methods to predict the outcome.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Experimental methods for estimating the impacts of text on human evaluation have been widely used in the social sciences. However, researchers in experimental settings are usually limited to testing a small number of pre-specified text treatments. While efforts to mine unstructured texts for features that causally affect outcomes have been ongoing in recent years, these models have primarily focused on the topics or specific words of text, which may not always be the mechanism of the effect. We connect these efforts with NLP interpretability techniques and present a method for flexibly discovering clusters of similar text phrases that are predictive of human reactions to texts using convolutional neural networks. When used in an experimental setting, this method can identify text treatments and their effects under certain assumptions. We apply the method to two datasets. The first enables direct validation of the model's ability to detect phrases known to cause the outcome. The second demonstrates its ability to flexibly discover text treatments with varying textual structures. In both cases, the model learns a greater variety of text treatments compared to benchmark methods, and these text features quantitatively meet or exceed the ability of benchmark methods to predict the outcome.

Related papers

• From Text to Treatment Effects: A Meta-Learning Approach to Handling Text-Based Confounding [7.5348062792]
  This paper examines the performance of meta-learners when confounding variables are expressed in text. We show that learners using pre-trained text representations of confounders achieve improved CATE estimates. Due to the entangled nature of the text embeddings, these models do not fully match the performance of meta-learners with perfect confounder knowledge.
  arXiv  Detail & Related papers  (2024-09-23T19:46:19Z)
• Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood [5.404146472517001]
  This study provides a new perspective by using the relative likelihood values instead of absolute ones. We propose a detection procedure with two classification methods, supervised and supervised-based, respectively. Our method can also reveal subtle differences between human and model languages, which find theoretical roots in psycholinguistics studies.
  arXiv  Detail & Related papers  (2024-06-28T12:28:52Z)
• Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams [49.3179290313959]
  This study explores the efficacy of seven text sampling methods designed to selectively fine-tune language models. We precisely assess the impact of these methods on fine-tuning the SBERT model using four different loss functions. Our findings indicate that Softmax loss and Batch All Triplets loss are particularly effective for text stream classification.
  arXiv  Detail & Related papers  (2024-03-18T23:41:52Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• An Information-Theoretic Approach for Detecting Edits in AI-Generated Text [7.013432243663526]
  We propose a method to determine whether a given article was written entirely by a generative language model or perhaps contains edits by a different author, possibly a human. We demonstrate the effectiveness of the method in detecting edits through extensive evaluations using real data. Our analysis raises several interesting research questions at the intersection of information theory and data science.
  arXiv  Detail & Related papers  (2023-08-24T12:49:21Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text [23.622347443796183]
  We study a more realistic setting where text begins as human-written and transitions to being generated by state-of-the-art neural language models. We show that, while annotators often struggle at this task, there is substantial variance in annotator skill and that given proper incentives, annotators can improve at this task over time.
  arXiv  Detail & Related papers  (2022-12-24T06:40:25Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• Be More with Less: Hypergraph Attention Networks for Inductive Text Classification [56.98218530073927]
  Graph neural networks (GNNs) have received increasing attention in the research community and demonstrated their promising results on this canonical task. Despite the success, their performance could be largely jeopardized in practice since they are unable to capture high-order interaction between words. We propose a principled model -- hypergraph attention networks (HyperGAT) which can obtain more expressive power with less computational consumption for text representation learning.
  arXiv  Detail & Related papers  (2020-11-01T00:21:59Z)
• Improving Text Generation with Student-Forcing Optimal Transport [122.11881937642401]
  We propose using optimal transport (OT) to match the sequences generated in training and testing modes. An extension is also proposed to improve the OT learning, based on the structural and contextual information of the text sequences. The effectiveness of the proposed method is validated on machine translation, text summarization, and text generation tasks.
  arXiv  Detail & Related papers  (2020-10-12T19:42:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.