Self-Adaptive Paraphrasing and Preference Learning for Improved Claim Verifiability

• URL: http://arxiv.org/abs/2412.11653v1
• Date: Mon, 16 Dec 2024 10:54:57 GMT
• Title: Self-Adaptive Paraphrasing and Preference Learning for Improved Claim Verifiability
• Authors: Amelie Wührl, Roman Klinger,
• Abstract summary: In fact-checking, structure and phrasing of claims critically influence a model's ability to predict verdicts accurately. We propose a self-adaptive approach to extract claims that is not reliant on labeled training data. We show that this novel setup extracts a claim paraphrase that is more verifiable than their original social media formulations.
• Score: 9.088303226909277
• License:
• Abstract: In fact-checking, structure and phrasing of claims critically influence a model's ability to predict verdicts accurately. Social media content in particular rarely serves as optimal input for verification systems, which necessitates pre-processing to extract the claim from noisy context before fact checking. Prior work suggests extracting a claim representation that humans find to be checkworthy and verifiable. This has two limitations: (1) the format may not be optimal for a fact-checking model, and (2), it requires annotated data to learn the extraction task from. We address both issues and propose a method to extract claims that is not reliant on labeled training data. Instead, our self-adaptive approach only requires a black-box fact checking model and a generative language model (LM). Given a tweet, we iteratively optimize the LM to generate a claim paraphrase that increases the performance of a fact checking model. By learning from preference pairs, we align the LM to the fact checker using direct preference optimization. We show that this novel setup extracts a claim paraphrase that is more verifiable than their original social media formulations, and is on par with competitive baselines. For refuted claims, our method consistently outperforms all baselines.

Related papers

• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference. Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Self-calibration for Language Model Quantization and Pruning [38.00221764773372]
  Quantization and pruning are fundamental approaches for model compression. In a post-training setting, state-of-the-art quantization and pruning methods require calibration data. We propose self-calibration as a solution.
  arXiv  Detail & Related papers  (2024-10-22T16:50:00Z)
• Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization [25.290462963681257]
  Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. They often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input.
  arXiv  Detail & Related papers  (2024-03-13T17:29:45Z)
• GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements [18.939302106382094]
  State-of-the-art language models can exhibit impressive reasoning refinement capabilities on math, science or coding tasks. But even the best models struggle to identify textitwhen and where to refine without access to external feedback. We propose Stepwise ORMs which are trained, only on synthetic data, to approximate the expected future reward of the optimal policy.
  arXiv  Detail & Related papers  (2024-02-13T20:16:29Z)
• Plausible Extractive Rationalization through Semi-Supervised Entailment Signal [29.67884478799914]
  We take a semi-supervised approach to optimize for the plausibility of extracted rationales. We adopt a pre-trained natural language inference (NLI) model and further fine-tune it on a small set of supervised rationales. We show that, by enforcing the alignment agreement between the explanation and answer in a question-answering task, the performance can be improved without access to ground truth labels.
  arXiv  Detail & Related papers  (2024-02-13T14:12:32Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Fine-tuning Language Models for Factuality [96.5203774943198]
  Large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations' In this work, we fine-tune language models to be more factual, without human labeling.
  arXiv  Detail & Related papers  (2023-11-14T18:59:15Z)
• Unsupervised Pretraining for Fact Verification by Language Model Distillation [4.504050940874427]
  We propose SFAVEL (Self-supervised Fact Verification via Language Model Distillation), a novel unsupervised pretraining framework. It distils self-supervised features into high-quality claim-fact alignments without the need for annotations. This is enabled by a novel contrastive loss function that encourages features to attain high-quality claim and evidence alignments.
  arXiv  Detail & Related papers  (2023-09-28T15:53:44Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)
• Generating Fact Checking Explanations [52.879658637466605]
  A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process. This paper provides the first study of how these explanations can be generated automatically based on available claim context. Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system.
  arXiv  Detail & Related papers  (2020-04-13T05:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.