Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit

• URL: http://arxiv.org/abs/2606.04274v1
• Date: Tue, 02 Jun 2026 22:58:59 GMT
• Title: Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit
• Abstract summary: We test an implicit assumption that scale and general capability are sufficient for nuanced classification of misinformation discourse.<n>We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs, and fine-tuned DistilBERT and RoBERTa.<n>In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice.
• Score: 5.437038443632133
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) become default tools for online information verification, an implicit assumption follows them: that scale and general capability are sufficient for nuanced classification of misinformation discourse. We test this assumption directly on 900 Reddit comments spanning three PolitiFact-verified misinformation claims (environment, health, immigration), labelled as belief (propagates the claim), fact-check (corrects it), or other. We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs (Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6), and fine-tuned DistilBERT and RoBERTa -- under universal and topic-specific label schemas. The assumption does not hold. Fine-tuned RoBERTa reaches 0.62 macro-$F_1$ against a best zero-shot result of 0.50 (Claude Haiku 4.5), at a fraction of the per-query cost; the supervised advantage is concentrated on the belief class, the implicit, affective category every zero-shot model under-detects. Scaling does not help: Llama-3-8B matches Llama-3-70B, and Claude Sonnet 4.6 underperforms the smaller Haiku under generic labels, collapsing belief detection to 0.17 and refusing outright on a subset of comments flagged as sensitive. This is a safety-alignment artefact, not a capacity limit. Label schema and topic jointly shape zero-shot performance, with the same model varying by more than 0.13 macro-$F_1$ across topics under matched labels. In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.

Related papers

• When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models [0.0]
  A language model, asked to verify 1,200 logical conclusions, answers YES every time.<n>linear probes on its hidden states read the correct verdict at 0.96 AUC.<n>We ask where the verdict is lost, and find the dominant failure is a single scalar.<n>The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset.
  arXiv  Detail & Related papers  (2026-09-04T00:26:45Z)
• Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge [46.547776251310694]
  This paper evaluates 5G domain understanding and fault analysis in a free-text generation format.<n>We evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks.
  arXiv  Detail & Related papers  (2026-08-21T12:09:51Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• HALLMARK: Diagnosing Three Failure Modes in LLM Citation Verifiers [30.02579334273979]
  GPTZero found 53 papers with hallucinated citations among NeurIPS 2025's accepted set.<n>No shared benchmark compares verifiers and gives detailed failure diagnostics.<n>Hallucination benchmark: 2,526 Bib entries spanning 14 types, three difficulty tiers, six diagnostic sub-tests per entry.
  arXiv  Detail & Related papers  (2026-07-20T13:21:27Z)
• Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks [0.11458853556386796]
  We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices.<n>We compare three LoRA-based fine-tuning paradigms on a unified setup across several Qwen3 models.<n> TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4.
  arXiv  Detail & Related papers  (2026-07-04T10:09:32Z)
• AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability [16.27940179594792]
  AdversaBench is an end-to-end red-teaming pipeline that mutates seed prompts with five structured operators.<n>We report experiments on 45 seeds across three categories: reasoning, instruction-following, and tool use.
  arXiv  Detail & Related papers  (2026-06-23T13:50:51Z)
• Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment [0.8699280339422538]
  Large language models are increasingly deployed as advisors whose objective is not aligned with the user's.<n>We turn the canonical Crawford-Sobel cheap-talk model into a benchmark for honesty under preference misalignment.<n>We find all four over-reveal relative to the most-informative equilibrium by 1.8 to 4.2x.
  arXiv  Detail & Related papers  (2026-05-31T21:30:43Z)
• Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety [0.0]
  We introduce a quality-diversity evolutionary framework that operates at the semantic level.<n>We maintain a diverse archive of attacks across behavioral dimensions.<n>The semantic representation produces interpretable attacks that reveal systematic, model-specific weaknesses.
  arXiv  Detail & Related papers  (2026-05-30T16:40:24Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models [0.0]
  LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures.<n>We benchmark three frontier LLMs on 100 natural-language questions over the Cleaned Contoso Retail dataset in ClickHouse.
  arXiv  Detail & Related papers  (2026-04-28T02:53:23Z)
• LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification [0.0]
  We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification.<n>The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC)<n>On a class-balanced test set, BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini)
  arXiv  Detail & Related papers  (2026-04-20T22:00:02Z)
• Beyond Hate: Differentiating Uncivil and Intolerant Speech in Multimodal Content Moderation [3.5637794289327847]
  We introduce a fine-grained annotation scheme that distinguishes two separable dimensions: incivility and intolerance.<n>We apply this scheme to 2,030 memes from the Hateful Memes dataset.<n>Our results show that fine-grained annotations complement existing coarse labels and, when used jointly, improve overall model performance.
  arXiv  Detail & Related papers  (2026-03-24T09:22:36Z)
• Probing for Knowledge Attribution in Large Language Models [45.47366023067617]
  Large language models (LLMs) often generate fluent but unfounded claims, or hallucinations.<n> Proper mitigation depends on knowing whether a model's answer is based on the prompt or its internal weights.<n>We show that a probe, a simple linear classifier trained on model hidden representations, can reliably predict contributive attribution.
  arXiv  Detail & Related papers  (2026-02-26T09:21:12Z)
• An Embarrassingly Simple Defense Against LLM Abliteration Attacks [47.347413305965006]
  A recent attack, termed abliteration, identifies and suppresses the single latent direction most responsible for refusal behavior.<n>We propose a defense that fundamentally alters how models express refusal.<n>Fine-tuning Llama-2-7B-Chat and Qwen2.5-Instruct yields models that maintain high refusal rates under abliteration.
  arXiv  Detail & Related papers  (2025-05-25T09:18:24Z)
• StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment [70.87096576708898]
  We propose StarFT, a framework for fine-tuning zero-shot models to enhance robustness by preventing them from learning spuriosity.<n>StarFT boosts both worst-group and average accuracy by 14.30% and 3.02%, respectively, in the Waterbirds group shift scenario.
  arXiv  Detail & Related papers  (2025-05-19T15:15:35Z)
• A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 [43.32593407341789]
  Despite promising performance on open-source large vision-language models, transfer-based targeted attacks often fail against closed-source commercial LVLMs.<n>We propose to refine semantic clarity by encoding explicit semantic details within local regions.<n>Our approach achieves success rates exceeding 90% on GPT-4.5, 4o, o1, significantly outperforming all prior state-of-the-art attack methods.
  arXiv  Detail & Related papers  (2025-03-13T17:59:55Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.