Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking

• URL: http://arxiv.org/abs/2608.23047v1
• Date: Mon, 24 Aug 2026 09:53:30 GMT
• Title: Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking
• Abstract summary: We introduce a graph-based framework for comparing human and LLM reasoning paths in scientific fact-checking.<n>For non-human- aligned LLM paths, we validate grounding in the cited study, relevance to the claim, and suf- ficiency for the verdict.<n>Results show distinct perfor- mance dimensions: Qwen3-32B has the lowest verdict failure rate, GPT-5 the highest human alignment, and Claude Opus 4.7 weak verdict prediction but often valid reasoning in success- ful cases.
• Score: 13.664000280311507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Misinformation that cites legitimate papers can be especially harmful when it distorts what those studies actually report. While existing automatic fact-checking systems based on large language models (LLMs) can assess whether a model assigns an Incorrect verdict and can gen- erate explanations for that decision, they typi- cally do not indicate whether the model follows the same reasoning path as human experts or arrives at the verdict through a different but still valid path. In this work, we introduce a graph- based framework (typed reasoning graph) for comparing human and LLM reasoning paths in scientific fact-checking. Building on prior work on fallacious reasoning in biomedical misinformation, MISSCIPLUS (Glockner et al., 2025), we model each explanation as a rea- soning graph that links the false claim to the relevant study context, study findings, fallacy- supporting premises, and fallacy labels. This representation enables one-to-one alignment of human and LLM reasoning at the level of fallacy-specific sub-graphs. For non-human- aligned LLM paths, we validate grounding in the cited study, relevance to the claim, and suf- ficiency for the verdict. Using 84 false claims from MISSCIPLUS, we evaluate GPT-5, Claude Opus 4.7, and Qwen3-32B across prompt and evidence settings. Results show distinct perfor- mance dimensions: Qwen3-32B has the lowest verdict failure rate, GPT-5 the highest human alignment, and Claude Opus 4.7 weak verdict prediction but often valid reasoning in success- ful cases

Related papers

• Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies [51.39569894108518]
  Large Language Models (LLMs) exhibit strong semantic capabilities, yet their resilience to manipulative linguistic patterns such as logical fallacies remains underexplored.<n>We introduce LoFa (Logical Fallacy), a benchmark for evaluating LLM robustness against fallacies.<n>We also propose Logical Fallacy Resistance at k (LFR@k), a metric that quantifies resistance to fallacious attacks.
  arXiv  Detail & Related papers  (2026-06-30T02:17:45Z)
• Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs [49.55219052565761]
  CausalPhys is a benchmark of over 3,000 carefully curated video- and image-based questions spanning four domains: Perception, Anticipation, Intervention, and Goal Orientation.<n>Each question is paired with an expert-annotated causal graph capturing object-attribute-event dependencies, enabling interpretable and fine-grained evaluation of causal understanding.<n>Building on this, we formulate a causal-graph-grounded metric that quantitatively measures how well a model's chain-of-thought reasoning aligns with the correct causal relations.<n>We propose Causal Rationale-informed Fine-Tuning (CRFT), which explicitly
  arXiv  Detail & Related papers  (2026-06-04T10:07:05Z)
• Abductive Reasoning with Syllogistic Forms in Large Language Models [4.33876606928699]
  Abduction is a process of drawing a minor premise from a major premise and conclusion.<n>This paper explores the accuracy of LLMs in abductive reasoning by converting a syllogistic dataset into one suitable for abduction.<n>It aims to investigate whether the state-of-the-art LLMs exhibit biases in abduction and to identify potential areas for improvement.
  arXiv  Detail & Related papers  (2026-03-06T16:06:25Z)
• Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification [56.51953062869371]
  DoVerifier is a symbolic verifier that checks whether causal expressions are derivable from a given causal graph using rules from do-calculus and probability theory.<n>Our evaluations on synthetic data and causal QA benchmarks show that DoVerifier more accurately captures semantic correctness of causal reasoning traces.
  arXiv  Detail & Related papers  (2026-01-29T03:22:58Z)
• Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning [0.0]
  We study logical reasoning in language models by asking whether their errors follow established human fallacy.<n>For each response, we judge logical reasoning and correctness when it matches an ETRpredicted fallacy.
  arXiv  Detail & Related papers  (2025-06-10T17:04:33Z)
• Do Large Language Models Reason Causally Like Us? Even Better? [7.749713014052951]
  Large language models (LLMs) have shown impressive capabilities in generating human-like text.<n>We compared causal reasoning in humans and four LLMs using tasks based on collider graphs.
  arXiv  Detail & Related papers  (2025-02-14T15:09:15Z)
• Grounding Fallacies Misrepresenting Scientific Publications in Evidence [84.32990746227385]
  We introduce MissciPlus, an extension of the fallacy detection dataset Missci.<n>MissciPlus pairs the real-world misrepresented evidence with incorrect claims, identical to the input to evidence-based fact-checking models.<n>Our findings show that current fact-checking models struggle to use misrepresented scientific passages to refute misinformation.
  arXiv  Detail & Related papers  (2024-08-23T03:16:26Z)
• Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
  Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Missci is a novel argumentation theoretical model for fallacious reasoning. We present Missci as a dataset to test the critical reasoning abilities of large language models.
  arXiv  Detail & Related papers  (2024-06-05T12:11:10Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• How susceptible are LLMs to Logical Fallacies? [5.723715910568911]
  We present LOGICOM, a diagnostic benchmark to assess the robustness of Large Language Models against logical fallacies. We use this benchmark to evaluate the performance of GPT-3.5 and GPT-4 using a dataset containing controversial topics. Our findings indicate that both GPT-3.5 and GPT-4 can adjust their opinion through reasoning.
  arXiv  Detail & Related papers  (2023-08-18T23:07:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.