Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?

• URL: http://arxiv.org/abs/2609.05088v1
• Date: Fri, 04 Sep 2026 12:40:39 GMT
• Title: Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?
• Abstract summary: We investigate an alternative standard designed to function despite realistic ambiguity.<n>We use a four-phase dialectical protocol grounded in Walton's theory of argumentation schemes and Govier's criteria for argument cogency.<n>It is adaptive to different frames of reasoning, extends beyond multiple-choice framing, and treats both the reasoning that precedes a verdict and its post-hoc justification.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI oversight methods rely on ground truth for validation, but what constitutes appropriate AI behavior is contested. This leaves evaluation of moral reasoning in LLMs and debate-based oversight implicitly avoiding realistic ambiguity. We investigate an alternative standard designed to function despite such ambiguity: structural quality of the defence a model can mount for its verdicts in response to critical questions, measured through a four-phase dialectical protocol grounded in Walton's theory of argumentation schemes and Govier's criteria for argument cogency. The protocol is adaptive to different frames of reasoning, extends beyond multiple-choice framing, and treats both the reasoning that precedes a verdict and its post-hoc justification. Across nine frontier models and 200 high-ambiguity MoralChoice items -- $6,778$ judge-scored cells, validated against $89.6\%$ inter-judge agreement on the binary failure judgment -- models defend their reasoning well above the rubric minimum on every dimension. Failure mass concentrates on grounds and sufficiency, and correlates with epistemic hedging rather than argument length. Reasoning is better defended than post-hoc justification, on every model and every Govier dimension. The scheme a model presents in its justification differs from the one it reasoned with on a substantial share of dilemmas ($\geq 20\%$ per model), despite value-based practical reasoning dominating both tracks. The protocol catches strictly indefensible defences (self-contradiction, false premises), and it surfaces difficulties in characterizing the role of retraction in AI alignment, suggesting a need for more situated evaluations.

Related papers

• A Contractualist Argumentation Framework for Moral Decision-Making [0.0]
  We propose a formal framework for moral decision-making grounded in Scanlon's contractualism.<n>We use ASPIC+, a structured argumentation framework, extended with value-based filtering to model how each agent's values determine which reasons are morally relevant.
  arXiv  Detail & Related papers  (2026-08-03T09:10:22Z)
• Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies [51.39569894108518]
  Large Language Models (LLMs) exhibit strong semantic capabilities, yet their resilience to manipulative linguistic patterns such as logical fallacies remains underexplored.<n>We introduce LoFa (Logical Fallacy), a benchmark for evaluating LLM robustness against fallacies.<n>We also propose Logical Fallacy Resistance at k (LFR@k), a metric that quantifies resistance to fallacious attacks.
  arXiv  Detail & Related papers  (2026-06-30T02:17:45Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• Collaborative Disagreement Resolution for Scalable Oversight [26.071237160687243]
  Disagreement resolution helps non-expert models identify the truth, achieving 62.1% judging accuracy compared to 49.2% for standard debate.<n>Our results provide encouraging empirical evidence for rethinking the scalable oversight protocol from adversarial persuasion to collaborative truth-seeking.
  arXiv  Detail & Related papers  (2026-06-02T12:55:49Z)
• Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning [50.352417879912515]
  Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities.<n>We propose Group Causal Counterfactual Policy Optimization to explicitly train LLMs to learn generalizable reasoning patterns.<n>We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust.
  arXiv  Detail & Related papers  (2026-02-06T08:03:11Z)
• Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models [72.4149653187766]
  We propose a Reasoner-Verifier framework named Adrialversa Reasoning RAG (ARR)<n>The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage.<n> Experiments on multiple benchmarks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2026-01-08T06:57:03Z)
• CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debate [3.2852123901391077]
  We develop a dual-agent debate framework for causal inference.<n>Agents attempt to persuade each other, challenging each other's logic.<n>We show that strong models can still benefit greatly from debate with weaker agents.
  arXiv  Detail & Related papers  (2025-11-28T03:19:35Z)
• Thinking Longer, Not Always Smarter: Evaluating LLM Capabilities in Hierarchical Legal Reasoning [11.255428720705204]
  We propose a framework that decomposes the process of identifying significant distinctions between cases into three-stage reasoning tasks.<n>Our framework models cases using factual predicates called factors, organizes them into a legal knowledge hierarchy, and defines verifiable rules for identifying distinctions.<n>We find that models consistently expend more computational resources on incorrect responses than correct ones, suggesting that "thinking longer" does not always mean "thinking smarter"
  arXiv  Detail & Related papers  (2025-10-09T18:15:28Z)
• "Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas [11.229443362516207]
  This study presents a comprehensive empirical evaluation of 14 leading large language models (LLMs)<n>We elicited 3,780 binary decisions and natural language justifications, enabling analysis along axes of decisional assertiveness, explanation answer consistency, public moral alignment, and sensitivity to ethically irrelevant cues.<n>We advocate for moral reasoning to become a primary axis in LLM alignment, calling for standardized benchmarks that evaluate not just what LLMs decide, but how and why.
  arXiv  Detail & Related papers  (2025-08-10T10:45:16Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.