Calibrating Small Language Models for Claim Check-Worthiness Detection

• URL: http://arxiv.org/abs/2608.30731v2
• Date: Tue, 01 Sep 2026 08:12:44 GMT
• Title: Calibrating Small Language Models for Claim Check-Worthiness Detection
• Abstract summary: NN-PPI is a pointwise extension of Prediction-Powered Inference that calibrates model predictions at inference time as a lightweight post-hoc layer.<n>NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model.
• Score: 7.856998585396422
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.

Related papers

• S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations [30.145244497712792]
  Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization.<n>We propose Selective Spectral Decay ($S2D$), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values.<n>We show that $S2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly.
  arXiv  Detail & Related papers  (2026-02-16T03:41:06Z)
• An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits [0.0]
  Post-training quantization reduces memory and computation but often degrades accuracy.<n>Pushing quantization to the ternary (2-bit) regime yields even larger savings but is notoriously unstable.<n>Our approach matches or surpasses more elaborate knowledge-distillation pipelines on standard language-modeling benchmarks.
  arXiv  Detail & Related papers  (2025-05-12T21:14:29Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training [17.381160429641316]
  Adaptive Sparse Trainer (AST) is a novel and efficient retraining framework tailored for semi-structured sparse models.<n>AST reduces the perplexity and zero-shot accuracy gap between dense and 2:4 semi-structured sparse models to 0.6 and 1.16%, respectively.
  arXiv  Detail & Related papers  (2024-07-30T06:33:44Z)
• Large Language Models Must Be Taught to Know What They Don't Know [97.90008709512921]
  We show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead.<n>We also investigate the mechanisms that enable reliable uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators.
  arXiv  Detail & Related papers  (2024-06-12T16:41:31Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses [14.77013588561901]
  We present LitCab, a lightweight calibration mechanism consisting of a single linear layer that takes the input text representation and predicts a bias term. For evaluation, we construct CaT, a benchmark consisting of eight text generation tasks, covering responses ranging from short phrases to paragraphs. We test LitCab with Llama2-7B, where it improves calibration across all tasks, reducing the average ECE score by as large as 30%.
  arXiv  Detail & Related papers  (2023-10-30T00:30:34Z)
• nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales [65.01417261415833]
  We present an approach to predict the pre-training loss based on our observations that Maximal Update Parametrization (muP) enables accurate fitting of scaling laws. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models.
  arXiv  Detail & Related papers  (2023-04-14T00:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.