Quantifying the Sources of Instability in LLM-Based Stance Analysis of Public Discourse

• URL: http://arxiv.org/abs/2607.10846v1
• Date: Sun, 12 Jul 2026 17:15:36 GMT
• Title: Quantifying the Sources of Instability in LLM-Based Stance Analysis of Public Discourse
• Abstract summary: Two distinct sources of instability can arise when preprocessing pipelines produce different outputs from the same input.<n>Using 256 YouTube interviews across 41 public figures from five domains, we compare two speaker-diarization pipelines and two measurement methods.<n>We find that preprocessing pipeline sensitivity is concentrated in speakers with limited video samples.
• Score: 4.093009863847175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computational social science increasingly relies on automated preprocessing pipelines -- speaker diarization, ASR transcript cleaning, sentence segmentation -- to convert raw media into analyzable text. When these pipelines produce different outputs from the same input, two distinct sources of instability can arise: the preprocessing pipeline itself (diarization method, segmentation rules) and the downstream measurement instrument (LLM annotation vs.\ keyword lexicon). Using 256 YouTube interviews across 41 public figures from five domains, we compare two speaker-diarization pipelines and two measurement methods, all targeting the coupling between affective valence and epistemic modality. We find that (1) preprocessing pipeline sensitivity is concentrated in speakers with limited video samples (N $\leq 5$); for the four best-sampled speakers (N $\geq 16$), the mean absolute pipeline-induced change in $r(\text{neg}, \text{emph})$ is only $0.13$; (2) cross-method disagreement is larger and more systematic -- the LLM and keyword-lexicon methods assign opposite coupling directions to several well-sampled speakers, even within the same preprocessing pipeline; and (3) aggregate valence proportions are highly stable ($|Δp(\text{neg})| < 6$pp) regardless of pipeline or method, masking both sources of instability. The contribution is a diagnostic framework that separates pipeline effects from measurement effects: researchers studying cross-dimensional relationships in interview data should verify that their conclusions are robust to both sources of variation, with particular attention to measurement method choice.

Related papers

• Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components [2.439525434311725]
  In domains relying on emphindirect observation, the learner observes a derivative representation produced by multi-stage measurement, reconstruction, and preprocessing pipelines.<n>We identify three failure modes arising from this frozen lens''<n>We call on the AI4Science community to make pipelines emphcomputable inference objects via domain-specific Computable Observation Frameworks.
  arXiv  Detail & Related papers  (2026-05-23T12:50:00Z)
• PARM: Pipeline-Adapted Reward Model [60.769414637325326]
  Reward models (RMs) are central to aligning large language models (LLMs) with human preferences, powering advanced decoding strategies.<n>While most prior work focuses on single-step generation, real-world applications increasingly adopt multi-stage pipelines.<n>We investigate this through code generation for optimization, constructing a pipeline that integrates reward models into both formulation and solution stages.
  arXiv  Detail & Related papers  (2026-04-20T14:29:08Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• $V_1$: Unifying Generation and Self-Verification for Parallel Reasoners [69.66089681814013]
  $V_$ is a framework that unifies generation and verification through efficient pairwise ranking.<n>$V_$-Infer improves Pass@1 by up to $10%$ over pointwise verification.<n>$V_$-PairRL achieves $7$--$9%$ test-time scaling gains over standard RL and pointwise joint training.
  arXiv  Detail & Related papers  (2026-03-04T17:22:16Z)
• A Dialectic Pipeline for Improving LLM Robustness [0.0]
  Methods such as fine-tuning on domain-specific data or the training of a separate textitad hoc verifier require demanding computational resources.<n>We propose a dialectic pipeline that preserves LLMs' generalization abilities while improving the quality of its answer via self-dialogue.<n>We find that our proposed dialectic pipeline is able to outperform by significative margins the standard model answers.
  arXiv  Detail & Related papers  (2026-01-28T14:42:49Z)
• DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation [24.086354908256293]
  textbfDVD is a single-sample detector that models the local output distribution induced by temperature sampling.<n>We construct the first benchmark for variant contamination across two domains Omni-MATH and SuperGPQA.<n>textbfDVD consistently outperforms perplexity-based, Min-$k$%++, edit-distance (CDD), and embedding-similarity baselines.
  arXiv  Detail & Related papers  (2026-01-08T12:48:40Z)
• FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation [1.6174863821322776]
  We introduce a benchmark designed to evaluate RAG pipelines as a whole.<n>We present a dataset of 93 questions designed to evaluate a pipeline's ability to ingest.<n>We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics.
  arXiv  Detail & Related papers  (2025-10-10T02:51:47Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Sound Event Classification in an Industrial Environment: Pipe Leakage Detection Use Case [3.9414768019101682]
  A multi-stage Machine Learning pipeline is proposed for pipe leakage detection in an industrial environment. The proposed pipeline applies multiple steps, each addressing the environment's challenges. The results show that the model produces excellent results with 99% accuracy and an F1-score of 0.93 and 0.9 for the respective datasets.
  arXiv  Detail & Related papers  (2022-05-05T15:26:22Z)
• Single-channel speech separation using Soft-minimum Permutation Invariant Training [60.99112031408449]
  A long-lasting problem in supervised speech separation is finding the correct label for each separated speech signal. Permutation Invariant Training (PIT) has been shown to be a promising solution in handling the label ambiguity problem. In this work, we propose a probabilistic optimization framework to address the inefficiency of PIT in finding the best output-label assignment.
  arXiv  Detail & Related papers  (2021-11-16T17:25:05Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.