A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2

• URL: http://arxiv.org/abs/2603.19253v1
• Date: Wed, 25 Feb 2026 11:17:24 GMT
• Title: A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2
• Abstract summary: This study presents a comprehensive evaluation of several state-of-the-art large language models (LLMs)<n>The evaluation incorporates advanced prompting strategies, including Chain-of- Thought prompting, prompt rephrasing, voting, and certainty-based classification.<n>The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (Args.me)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Argument mining (AM) is an interdisciplinary research field focused on the automatic identification and classification of argumentative components, such as claims and premises, and the relationships between them. Recent advances in large language models (LLMs) have significantly improved the performance of argument classification compared to traditional machine learning approaches. This study presents a comprehensive evaluation of several state-of-the-art LLMs, including GPT-5.2, Llama 4, and DeepSeek, on large publicly available argument classification corpora such as Args.me and UKP. The evaluation incorporates advanced prompting strategies, including Chain-of- Thought prompting, prompt rephrasing, voting, and certainty-based classification. Both quantitative performance metrics and qualitative error analysis are conducted to assess model behavior. The best-performing model in the study (GPT-5.2) achieves a classification accuracy of 78.0% (UKP) and 91.9% (Args.me). The use of prompt rephrasing, multi-prompt voting, and certainty estimation further improves classification performance and robustness. These techniques increase the accuracy and F1 metric of the models by typically a few percentage points (from 2% to 8%). However, qualitative analysis reveals systematic failure modes shared across models, including instabilities with respect to prompt formulation, difficulties in detecting implicit criticism, interpreting complex argument structures, and aligning arguments with specific claims. This work contributes the first comprehensive evaluation that combines quantitative benchmarking and qualitative error analysis on multiple argument mining datasets using advanced LLM prompting strategies.

Related papers

• Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals [75.25256166997414]
  We introduce Metacognition-as-Reward (MaR), a metacognition-inspired RL framework.<n>MaR guides reasoning through two general process dimensions.<n>MaR consistently improves model performance, achieving up to a 7.7% gain over the base model.
  arXiv  Detail & Related papers  (2026-05-22T08:54:37Z)
• Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams [1.6955250288861923]
  Student responses in STEM assessments are often handwritten and combine symbolic expressions, calculations, and diagrams.<n>Recent advances in large language models (LLMs) have increased attention to AI-assisted scoring.<n>This study examined the reliability of AI-assisted scoring of undergraduate physics constructed responses using GPT-4o.
  arXiv  Detail & Related papers  (2026-04-14T03:04:44Z)
• Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators [2.952559770088323]
  Large language models (LLMs) offer new opportunities for automated research evaluation based on textual content.<n>This study examines whether LLMs can support post-publication peer review tasks by benchmarking their outputs against expert judgments and citation-based indicators.
  arXiv  Detail & Related papers  (2026-03-27T11:32:18Z)
• Enhancing the QA Model through a Multi-domain Debiasing Framework [1.7802147489386633]
  This study evaluates the ELECTRA-small model on the Stanford Question Answering dataset (SQuAD) v1.1 and adversarial datasets AddSent and AddOneSent.<n>We develop a multi-domain debiasing framework incorporating knowledge distillation, debiasing techniques, and domain expansion.
  arXiv  Detail & Related papers  (2026-01-01T08:39:07Z)
• What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation [67.47463575774388]
  We decompose reasoning quality into two dimensions: relevance and coherence.<n>To measure these aspects reliably, we introduce causal stepwise evaluation (CaSE)<n>We show that curating training data with CaSE-evaluated relevance and coherence directly improves final task performance.
  arXiv  Detail & Related papers  (2025-10-23T14:30:37Z)
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs [55.20230501807337]
  We present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework.<n>We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset.
  arXiv  Detail & Related papers  (2025-08-15T10:32:50Z)
• A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1 [0.0]
  Large language models (LLMs) have enhanced the efficiency of analyzing and extracting argument semantics.<n>This paper presents a study of a selection of LLM's, using diverse datasets such as Args.me and UKP.<n>The results indicate that ChatGPT-4o outperforms the others in the argument classification benchmarks.
  arXiv  Detail & Related papers  (2025-07-11T14:23:40Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Efficient argument classification with compact language models and ChatGPT-4 refinements [0.0]
  This paper presents comparative studies between a few deep learning-based models in argument mining. The main novelty of this paper is the ensemble model which is based on BERT architecture and ChatGPT-4 as fine tuning model. The presented results show that BERT+ChatGPT-4 outperforms the rest of the models including other Transformer-based and LSTM-based models.
  arXiv  Detail & Related papers  (2024-03-20T16:24:10Z)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation [60.65820977963331]
  We introduce a novel evaluation paradigm for Large Language Models (LLMs) This paradigm shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-12-28T15:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.