A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection

• URL: http://arxiv.org/abs/2604.26319v1
• Date: Wed, 29 Apr 2026 06:02:32 GMT
• Title: A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection
• Abstract summary: Stance detection identifies the attitude of a text author toward a given target.<n>Recent studies have explored various LLM-based strategies for this task, from zero-shot prompting to multi-agent debate.<n>We conduct a systematic comparison that evaluates five methods across two categories -- prompt-based inference (Direct Prompting, Auto-CoT, StSQA) and agent-based debate (COLA, MPRF)<n>Our experiments yield several findings. First, on all models with complete results, the best prompt-based method outperforms the best agent-based method, while agent methods require 7 to 12 times more API calls per
• Score: 11.458166456268103
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Stance detection identifies the attitude of a text author toward a given target. Recent studies have explored various LLM-based strategies for this task, from zero-shot prompting to multi-agent debate. However, existing works differ in data splits, base models, and evaluation protocols, making fair comparison difficult. We conduct a systematic comparison that evaluates five methods across two categories -- prompt-based inference (Direct Prompting, Auto-CoT, StSQA) and agent-based debate (COLA, MPRF) -- on four datasets with 14 subtasks, using 15 LLMs from six model families with parameter sizes from 7B to 72B+. Our experiments yield several findings. First, on all models with complete results, the best prompt-based method outperforms the best agent-based method, while agent methods require 7 to 12 times more API calls per sample. Second, model scale has a larger impact on performance than method choice, with gains plateauing around 32B. Third, reasoning-enhanced models (DeepSeek-R1) do not consistently outperform general models of the same size on this task.

Related papers

• Beyond Exact Match: How Evaluation Methodology Dominates Model Choice in LLM-Based Product Attribute Extraction [0.0]
  We compare four prompting strategies -- zero-shot, few-shot, schema-guided, and definition-augmented -- across two production-grade large language models.<n>We find that evaluation methodology produces variance approximately 23 times larger than model choice.<n>We conclude that for production attribute extraction pipelines, evaluation methodology and data quality dominate the impact of model selection and prompt engineering.
  arXiv  Detail & Related papers  (2026-07-24T23:25:16Z)
• Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models [0.0]
  We present a controlled study of multi-hop contextual reasoning in large language models.<n>We show that multi-agent systems show the inverse pattern, achieving up to 80% on reasoning tasks where rule-based methods fail.
  arXiv  Detail & Related papers  (2026-01-06T20:18:55Z)
• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• A Multi-To-One Interview Paradigm for Efficient MLLM Evaluation [63.76972456980632]
  We propose a multi-to-one interview paradigm for efficient MLLM evaluation.<n>Our framework consists of (i) a two-stage interview strategy with pre-interview and formal interview phases, (ii) dynamic adjustment of weights to ensure fairness, and (iii) an adaptive mechanism for question difficulty-level chosen.
  arXiv  Detail & Related papers  (2025-09-18T12:07:40Z)
• Multi-Intent Recognition in Dialogue Understanding: A Comparison Between Smaller Open-Source LLMs [6.6893292050680655]
  We use the MultiWOZ 2.1 dataset, a benchmark in the dialogue system domain, to investigate the efficacy of three popular open-source pre-trained LLMs.<n>We perform the classification task in a few-shot setup, giving 20 examples in the prompt with some instructions.<n>We find the Mistral-7B-v0.1 outperforms two other generative models on 11 intent classes out of 14 in terms of F-Score.
  arXiv  Detail & Related papers  (2025-09-12T07:10:55Z)
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs [55.20230501807337]
  We present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework.<n>We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset.
  arXiv  Detail & Related papers  (2025-08-15T10:32:50Z)
• CTTS: Collective Test-Time Scaling [58.564620942591866]
  Test-time scaling (TTS) has emerged as a promising, training-free approach for enhancing large language model (LLM) performance.<n>We introduce Collective Test-Time Scaling (CTTS) to overcome the dominant single test-time scaling (STTS) paradigm.<n>CTTS-MM is a novel framework that operationalizes multi-agent and multi-reward collaboration.
  arXiv  Detail & Related papers  (2025-08-05T11:19:08Z)
• What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models [3.113304966059062]
  Large Language Models (LLMs) are increasingly applied for Process Modeling (PMo) tasks such as Process Model Generation (PMG)<n>Process Model Representations (PMRs) differ widely in structure, complexity, and usability, and have never been systematically compared.<n>This paper presents the first empirical study that evaluates multiple PMRs in the context of PMo with LLMs.
  arXiv  Detail & Related papers  (2025-07-15T14:26:50Z)
• Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications [0.7124971549479361]
  This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification.<n>We determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability.
  arXiv  Detail & Related papers  (2025-05-20T21:12:58Z)
• VisualPRM: An Effective Process Reward Model for Multimodal Reasoning [76.35753243272521]
  We introduce VisualPRM, which improves the reasoning abilities of existing Multimodal Large Language Models (MLLMs)<n>Our model achieves a 5.9-point improvement across seven multimodal reasoning benchmarks.<n>For the evaluation of multimodal PRMs, we propose VisualProcessBench, a benchmark with human-annotated step-wise correctness labels.
  arXiv  Detail & Related papers  (2025-03-13T12:03:37Z)
• Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models [56.256069117502385]
  Chain of Thought (CoT) approaches can be used to enhance the capability of Large Language Models (LLMs) on complex reasoning tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored. We introduce a novel approach that addresses this challenge by using retrieval mechanisms to automatically select demonstration examples.
  arXiv  Detail & Related papers  (2023-12-04T08:07:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.