Contextualized Prompting For Stance Detection On Social Media

• URL: http://arxiv.org/abs/2606.06022v1
• Date: Thu, 04 Jun 2026 11:14:44 GMT
• Title: Contextualized Prompting For Stance Detection On Social Media
• Abstract summary: Stance detection on social media is challenging due to short, noisy, and context-dependent language.<n>We investigate the impact of incorporating contextual information into large language models.<n>We find that integrating contextual information improves performance, but only under specific conditions.
• Score: 41.666445939998454
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stance detection on social media is challenging due to short, noisy, and context-dependent language. While large language models (LLMs) show zero-shot generalization, they are typically prompted without contextual information, which limits their ability to interpret ambiguous posts. In this work, we systematically investigate the impact of incorporating real-world (e.g., user biographies), derived (e.g., political party), and LLM-generated (e.g., target descriptions) contextual features into zero-shot prompting for stance detection on Twitter. Our evaluation spans four benchmark datasets, including a new high-quality German Twitter stance dataset. Across multiple LLMs, we find that integrating contextual information improves performance, but only under specific conditions. LLM-generated target descriptions consistently enhance accuracy, while other user metadata has mixed or even detrimental effects. Notably, we show that the inclusion of other tweets by the same user, often beneficial in supervised learning, can impair performance due to input noise. Our qualitative analysis reveals that LLMs struggle to distinguish task-specific useful information from irrelevant context. Our findings highlight both the promise and challenges of prompting with context information in noisy real-world settings. We publish code and data at this \href{https://github.com/tilmanbeck/stance-context-twitter}{page}.

Related papers

• Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models [67.09110757873142]
  We introduce Text2DistBench, a reading comprehension benchmark for evaluating LLMs' ability to infer distributional knowledge from natural language.<n>Built from real-world YouTube comments about movie and music entities, the benchmark provides models with entity metadata and associated comments.<n>To support reliable and long-term evaluation, the construction pipeline of Text2DistBench is fully automated and continuously updated to incorporate newly emerging entities.
  arXiv  Detail & Related papers  (2026-03-13T19:26:08Z)
• PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions [55.95282725491425]
  PoSh is a metric for detailed image description that uses scene graphs as structured rubrics to guide LLMs-as-a-Judge.<n>PoSh is replicable, interpretable and a better proxy for human raters than existing metrics.<n>We show that PoSh achieves stronger correlations with the human judgments in DOCENT than the best open-weight alternatives.
  arXiv  Detail & Related papers  (2025-10-21T20:30:20Z)
• Is External Information Useful for Stance Detection with LLMs? [1.0610707915233728]
  Prior work suggests that the use of external information, e.g., excerpts from Wikipedia, improves stance detection performance.<n>We conduct a systematic evaluation on how Wikipedia and web search external information can affect stance detection.<n>We find that such information degrades performance in most cases, with macro F1 scores dropping by up to 27.9%.
  arXiv  Detail & Related papers  (2025-07-02T09:53:41Z)
• Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data [2.812898346527047]
  This study investigates the capabilities of large language models (LLMs) for zero-shot and few-shot annotation of social media posts in Russian and Ukrainian.<n>To evaluate the effectiveness of these models, their annotations are compared against a gold standard set of human double-annotated labels.<n>The study explores the unique patterns of errors and disagreements exhibited by each model, offering insights into their strengths, limitations, and cross-linguistic adaptability.
  arXiv  Detail & Related papers  (2025-05-15T13:10:47Z)
• LLM-based Semantic Augmentation for Harmful Content Detection [5.954202581988127]
  This paper introduces an approach that prompts large language models to clean noisy text and provide context-rich explanations.<n>We evaluate on the SemEval 2024 multi-label Persuasive Meme dataset and validate on the Google Jigsaw toxic comments and Facebook hateful memes datasets.<n>Our results reveal that zero-shot LLM classification underperforms on these high-context tasks compared to supervised models.
  arXiv  Detail & Related papers  (2025-04-22T02:59:03Z)
• Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context [12.781022584125925]
  We construct a novel, controlled contrastive dataset designed to test whether LLMs can effectively use context to disambiguate idiomatic meaning.<n>Our findings reveal that LLMs often fail to resolve idiomaticity when it is required to attend to the surrounding context.<n>We make our code and dataset publicly available.
  arXiv  Detail & Related papers  (2024-10-21T14:47:37Z)
• LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements [59.71218039095155]
  Task of reading comprehension (RC) provides a primary means to assess language models' natural language understanding (NLU) capabilities.<n>If the context aligns with the models' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from internal information.<n>To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities.
  arXiv  Detail & Related papers  (2024-04-09T13:08:56Z)
• HICL: Hashtag-Driven In-Context Learning for Social Media Natural Language Understanding [15.743523533234224]
  In this paper, we propose a novel hashtag-driven in-context learning framework for natural language understanding on social media. Our objective is to enable a model #Encoder to incorporate topic-related semantic information, which allows it to retrieve topic-related posts. For empirical studies, we collected 45M tweets to set up an in-context NLU benchmark, and the experimental results on seven downstream tasks show that HICL substantially advances the previous state-of-the-art results.
  arXiv  Detail & Related papers  (2023-08-19T11:31:45Z)
• SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks [88.4408774253634]
  Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community. There are not nearly as many SLU task benchmarks, and many of the existing ones use data that is not freely available to all researchers. Recent work has begun to introduce such benchmark for several tasks.
  arXiv  Detail & Related papers  (2022-12-20T18:39:59Z)
• Local-Global Video-Text Interactions for Temporal Grounding [77.5114709695216]
  This paper addresses the problem of text-to-video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query. We tackle this problem using a novel regression-based model that learns to extract a collection of mid-level features for semantic phrases in a text query. The proposed method effectively predicts the target time interval by exploiting contextual information from local to global.
  arXiv  Detail & Related papers  (2020-04-16T08:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.