The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance

• URL: http://arxiv.org/abs/2604.03237v1
• Date: Sat, 31 Jan 2026 20:23:37 GMT
• Title: The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance
• Abstract summary: We identify a Persuasion Paradox: fluent explanations systematically increase user confidence and reliance on AI without reliably improving, and in some cases undermining, task accuracy.<n>Our findings demonstrate that commonly used subjective metrics such as trust, confidence, and perceived clarity are poor predictors of human-AI team performance.
• Score: 16.560095082937874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While natural-language explanations from large language models (LLMs) are widely adopted to improve transparency and trust, their impact on objective human-AI team performance remains poorly understood. We identify a Persuasion Paradox: fluent explanations systematically increase user confidence and reliance on AI without reliably improving, and in some cases undermining, task accuracy. Across three controlled human-subject studies spanning abstract visual reasoning (RAVEN matrices) and deductive logical reasoning (LSAT problems), we disentangle the effects of AI predictions and explanations using a multi-stage reveal design and between-subjects comparisons. In visual reasoning, LLM explanations increase confidence but do not improve accuracy beyond the AI prediction alone, and substantially suppress users' ability to recover from model errors. Interfaces exposing model uncertainty via predicted probabilities, as well as a selective automation policy that defers uncertain cases to humans, achieve significantly higher accuracy and error recovery than explanation-based interfaces. In contrast, for language-based logical reasoning tasks, LLM explanations yield the highest accuracy and recovery rates, outperforming both expert-written explanations and probability-based support. This divergence reveals that the effectiveness of narrative explanations is strongly task-dependent and mediated by cognitive modality. Our findings demonstrate that commonly used subjective metrics such as trust, confidence, and perceived clarity are poor predictors of human-AI team performance. Rather than treating explanations as a universal solution, we argue for a shift toward interaction designs that prioritize calibrated reliance and effective error recovery over persuasive fluency.

Related papers

• Human Decision-Making with Persuasive and Narrative LLM Explanations [11.224478276578745]
  Large language models (LLMs) have the potential to aid and improve human decision-making in classification tasks.<n>Prior work has demonstrated that people generally find AI narrative explanations to be understandable, trustworthy, and convincing for changing beliefs and opinions.<n>Here we evaluate the impact of narrative explanations on objective human decision-making performance.
  arXiv  Detail & Related papers  (2026-05-22T17:25:02Z)
• Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs [68.58207076756237]
  We introduce ProCauEval, a perturbation-based evaluation protocol that shifts from outcome assessment to mechanism diagnosis.<n>We find that models faithfully perceive video content yet systematically underexploit it during causal reasoning.<n>We propose Anti-Distillation Policy Optimization (ADPO), a reinforcement learning framework built on negative teacher alignment.
  arXiv  Detail & Related papers  (2026-05-10T08:48:58Z)
• When AI Persuades: Adversarial Explanation Attacks on Human Trust in AI-Assisted Decision Making [7.170587130743388]
  Large Language Models generate fluent natural-language explanations that shape how users perceive and trust AI outputs.<n>We introduce adversarial explanation attacks (AEAs), where an attacker manipulates the framing of LLM-generated explanations to modulate human trust in incorrect outputs.<n>This is the first systematic security study that treats explanations as an adversarial cognitive channel and quantifies their impact on human trust in AI-assisted decision making.
  arXiv  Detail & Related papers  (2026-02-03T20:42:44Z)
• The AI Cognitive Trojan Horse: How Large Language Models May Bypass Human Epistemic Vigilance [0.0]
  Large language model (LLM)-based conversational AI systems present a challenge to human cognition.<n>This paper proposes that a significant epistemic risk from conversational AI may lie not in inaccuracy or intentional deception, but in something more fundamental.
  arXiv  Detail & Related papers  (2026-01-11T22:28:56Z)
• Cognitive Inception: Agentic Reasoning against Visual Deceptions by Injecting Skepticism [81.39177645864757]
  We propose textbfInception, a fully reasoning-based agentic reasoning framework to conduct authenticity verification by injecting skepticism.<n>To the best of our knowledge, this is the first fully reasoning-based framework against AIGC visual deceptions.
  arXiv  Detail & Related papers  (2025-11-21T05:13:30Z)
• Exploring and Exploiting the Inherent Efficiency within Large Reasoning Models for Self-Guided Efficiency Enhancement [101.77467538102924]
  Large reasoning models (LRMs) exhibit overthinking, which hinders efficiency and inflates inference cost.<n>We propose two lightweight methods to enhance LRM efficiency.<n>First, we introduce Efficiency Steering, a training-free activation steering technique that modulates reasoning behavior via a single direction.<n>Second, we develop Self-Rewarded Efficiency RL, a reinforcement learning framework that dynamically balances task accuracy and brevity.
  arXiv  Detail & Related papers  (2025-06-18T17:18:12Z)
• Learning to Generate and Evaluate Fact-checking Explanations with Transformers [10.970249299147866]
  Research contributes to the field of Explainable Artificial Antelligence (XAI) We develop transformer-based fact-checking models that contextualise and justify their decisions by generating human-accessible explanations. We emphasise the need for aligning Artificial Intelligence (AI)-generated explanations with human judgements.
  arXiv  Detail & Related papers  (2024-10-21T06:22:51Z)
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust [51.542856739181474]
  We show how different natural language expressions of uncertainty impact participants' reliance, trust, and overall task performance. We find that first-person expressions decrease participants' confidence in the system and tendency to agree with the system's answers, while increasing participants' accuracy. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters.
  arXiv  Detail & Related papers  (2024-05-01T16:43:55Z)
• Tuning-Free Accountable Intervention for LLM Deployment -- A Metacognitive Approach [55.613461060997004]
  Large Language Models (LLMs) have catalyzed transformative advances across a spectrum of natural language processing tasks. We propose an innovative textitmetacognitive approach, dubbed textbfCLEAR, to equip LLMs with capabilities for self-aware error identification and correction.
  arXiv  Detail & Related papers  (2024-03-08T19:18:53Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• A Study on Multimodal and Interactive Explanations for Visual Question Answering [3.086885687016963]
  We evaluate multimodal explanations in the setting of a Visual Question Answering (VQA) task. Results indicate that the explanations help improve human prediction accuracy, especially in trials when the VQA system's answer is inaccurate. We introduce active attention, a novel method for evaluating causal attentional effects through intervention by editing attention maps.
  arXiv  Detail & Related papers  (2020-03-01T07:54:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.