Chain-of-Thought Embeddings for Stance Detection on Social Media

• URL: http://arxiv.org/abs/2310.19750v1
• Date: Mon, 30 Oct 2023 17:18:10 GMT
• Title: Chain-of-Thought Embeddings for Stance Detection on Social Media
• Authors: Joseph Gatto, Omar Sharif, Sarah Masud Preum
• Abstract summary: Chain-of-Thought (COT) prompting has recently been shown to improve performance on stance detection tasks. Our study introduces COT Embeddings which improve COT performance on stance detection tasks. Our model achieves SOTA performance on multiple stance detection datasets collected from social media.
• Score: 3.5128547933798275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stance detection on social media is challenging for Large Language Models (LLMs), as emerging slang and colloquial language in online conversations often contain deeply implicit stance labels. Chain-of-Thought (COT) prompting has recently been shown to improve performance on stance detection tasks -- alleviating some of these issues. However, COT prompting still struggles with implicit stance identification. This challenge arises because many samples are initially challenging to comprehend before a model becomes familiar with the slang and evolving knowledge related to different topics, all of which need to be acquired through the training data. In this study, we address this problem by introducing COT Embeddings which improve COT performance on stance detection tasks by embedding COT reasonings and integrating them into a traditional RoBERTa-based stance detection pipeline. Our analysis demonstrates that 1) text encoders can leverage COT reasonings with minor errors or hallucinations that would otherwise distort the COT output label. 2) Text encoders can overlook misleading COT reasoning when a sample's prediction heavily depends on domain-specific patterns. Our model achieves SOTA performance on multiple stance detection datasets collected from social media.

Related papers

• A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning [9.786907179872815]
  The potential of vision and language remains underexplored in face forgery detection. There is a need for a methodology that converts face forgery detection to a Visual Question Answering (VQA) task. We propose a multi-staged approach that diverges from the traditional binary decision paradigm to address this gap.
  arXiv  Detail & Related papers  (2024-10-01T08:16:40Z)
• A Challenge Dataset and Effective Models for Conversational Stance Detection [26.208989232347058]
  We introduce a new multi-turn conversation stance detection dataset (called textbfMT-CSD) We propose a global-local attention network (textbfGLAN) to address both long and short-range dependencies inherent in conversational data. Our dataset serves as a valuable resource to catalyze advancements in cross-domain stance detection.
  arXiv  Detail & Related papers  (2024-03-17T08:51:01Z)
• Chain of Thought Explanation for Dialogue State Tracking [52.015771676340016]
  Dialogue state tracking (DST) aims to record user queries and goals during a conversational interaction. We propose a model named Chain-of-Thought-Explanation (CoTE) for the DST task. CoTE is designed to create detailed explanations step by step after determining the slot values.
  arXiv  Detail & Related papers  (2024-03-07T16:59:55Z)
• Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors [57.7003399760813]
  We explore advanced Large Language Models (LLMs) and their specialized variants, contributing to this field in several ways. We uncover a significant correlation between topics and detection performance. These investigations shed light on the adaptability and robustness of these detection methods across diverse topics.
  arXiv  Detail & Related papers  (2023-12-20T10:53:53Z)
• Noise-Robust Dense Retrieval via Contrastive Alignment Post Training [89.29256833403167]
  Contrastive Alignment POst Training (CAPOT) is a highly efficient finetuning method that improves model robustness without requiring index regeneration. CAPOT enables robust retrieval by freezing the document encoder while the query encoder learns to align noisy queries with their unaltered root. We evaluate CAPOT noisy variants of MSMARCO, Natural Questions, and Trivia QA passage retrieval, finding CAPOT has a similar impact as data augmentation with none of its overhead.
  arXiv  Detail & Related papers  (2023-04-06T22:16:53Z)
• How would Stance Detection Techniques Evolve after the Launch of ChatGPT? [5.756359016880821]
  A new pre-trained language model chatGPT was launched on Nov 30, 2022. ChatGPT can achieve SOTA or similar performance for commonly used datasets including SemEval-2016 and P-Stance. ChatGPT has the potential to be the best AI model for stance detection tasks in NLP.
  arXiv  Detail & Related papers  (2022-12-30T05:03:15Z)
• Catch Me If You Can: Deceiving Stance Detection and Geotagging Models to Protect Privacy of Individuals on Twitter [3.928604516640069]
  We ground our investigation in two exposure-risky tasks, stance detection and geotagging. We explore a variety of simple techniques for modifying text, such as inserting typos in salient words, paraphrasing, and adding dummy social media posts. We find that typos have minimal impact on state-of-the-art geotagging models due to their increased reliance on social networks.
  arXiv  Detail & Related papers  (2022-07-23T11:55:18Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)
• Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation [62.943925893616196]
  We study the possibilities of building a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC) CTC's success on translation is counter-intuitive due to its monotonicity assumption, so we analyze its reordering capability. Our analysis shows that transformer encoders have the ability to change the word order.
  arXiv  Detail & Related papers  (2021-05-11T07:48:45Z)
• Time to Transfer: Predicting and Evaluating Machine-Human Chatting Handoff [36.62707486132739]
  We introduce the Machine-Human Chatting Handoff (MHCH), which enables human-algorithm collaboration. To detect the normal/transferable utterances, we propose a Difficulty-Assisted Matching Inference (DAMI) network. A matching inference mechanism is introduced to capture the contextual matching features.
  arXiv  Detail & Related papers  (2020-12-14T15:02:08Z)
• Exploiting Unsupervised Data for Emotion Recognition in Conversations [76.01690906995286]
  Emotion Recognition in Conversations (ERC) aims to predict the emotional state of speakers in conversations. The available supervised data for the ERC task is limited. We propose a novel approach to leverage unsupervised conversation data.
  arXiv  Detail & Related papers  (2020-10-02T13:28:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.