Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference

• URL: http://arxiv.org/abs/2608.03388v1
• Date: Tue, 04 Aug 2026 09:39:24 GMT
• Title: Don't Let Me Ask for It: LLMs Show Deficiencies in Active Multi-Turn Information Acquisition for Abductive Inference
• Abstract summary: Abductive reasoning requires forming hypotheses that explain observed evidence and revising them as new evidence becomes available.<n>We introduce Alien Abduction, an interactive probe for studying these behaviours under different interaction modes.
• Score: 15.768100289136392
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Abductive reasoning requires forming hypotheses that explain observed evidence and revising them as new evidence becomes available. While large language models (LLMs) are often evaluated on whether they solve abductive reasoning tasks correctly, less is known about how they acquire evidence, update their hypotheses, and decide when to stop. We introduce Alien Abduction game, an interactive probe for studying these behaviours under different interaction modes. The modes vary in whether evidence is provided upfront or across turns, and whether queries are selected by the model or examples are provided by the oracle. Across models, providing evidence upfront leads to higher success rates than distributing it across turns. In multi-turn settings, some models commit before using the available evidence, while others exhaust the turn budget without converging. Models also achieve higher success rates when examples are provided by the oracle than when they select their own queries, although their final hypotheses are more consistent with the evidence they selected. These findings suggest that models may form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives, and may struggle to validate and refine their hypotheses or determine when to stop.

Related papers

• Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts [49.97842835825998]
  Existing benchmarks conflate two forms of evidence within a single modality: perceptual signals and propositional signals.<n>We introduce Tri-PvP, a benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form.<n>We find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias.
  arXiv  Detail & Related papers  (2026-09-05T10:33:58Z)
• DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark [6.835919520017402]
  Defeasible reasoning is a type of reasoning where inferences are drawn from plausible current evidence, but can be retracted upon the introduction of newer evidence.<n>We introduce DeReLab, a generative framework that produces multi-turn belief-updating conversations from parameterized graph structures.<n>We evaluate nine open and proprietary large language models and find that nearly all exhibit a systematic tendency to accept congruent evidence while resisting incongruent updates.
  arXiv  Detail & Related papers  (2026-08-31T08:10:29Z)
• EVAR: Evidence-Validated Hypothesis Admission for Budget-Aware Narrative Reasoning [6.590648135605556]
  EVAR is an evidence-validated hypothesis admission framework for budget-aware narrative reasoning.<n>We show that EVAR improves both task performance and evidence faithfulness while maintaining controllable inference cost.
  arXiv  Detail & Related papers  (2026-08-30T15:07:27Z)
• Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoning [38.63002496590365]
  We propose a framework for improving reliability in knowledge-intensive reasoning through equivariance-based abstention.<n>A key contributing factor is that entity mentions in context activate memorised associations, causing models to generate plausible responses ungrounded in evidence.<n>We construct multiple worlds via typed substitutions of the original input that preserve relational structure while reducing parametric priors.
  arXiv  Detail & Related papers  (2026-08-28T07:33:29Z)
• FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games [2.959715295394151]
  We introduce FALSIFYBENCH, an evaluation framework for hypothesis-driven reasoning inspired by the classic Wason 2-4-6 task.<n>This task captures key elements of scientific reasoning: hypothesis generation, evidence gathering, and belief revision in response to both confirming and disconfirming evidence.<n>Our evaluation of 12 LLMs across model families and scales shows that reasoning models are generally stronger scientific reasoners than instruction-tuned models, although no model comes close to optimal performance.
  arXiv  Detail & Related papers  (2026-06-03T11:33:17Z)
• VITED: Video Temporal Evidence Distillation [49.38292490256531]
  We investigate complex video question answering via chain-of-evidence reasoning.<n>Models struggle with multi-step reasoning as they uniformly sample a fixed number of frames.<n>We propose a framework to enhance existing VideoQA datasets with evidence reasoning chains.
  arXiv  Detail & Related papers  (2025-03-17T06:30:02Z)
• Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty [47.95943057892318]
  Quantifying uncertainty in black-box LLMs is vital for reliable responses and scalable oversight.<n>We introduce DiverseAgentEntropy, a novel, theoretically-grounded method employing multi-agent interaction for uncertainty estimation.
  arXiv  Detail & Related papers  (2024-12-12T18:52:40Z)
• A Belief Model for Conflicting and Uncertain Evidence -- Connecting Dempster-Shafer Theory and the Topology of Evidence [8.295493796476766]
  We propose a new model for measuring degrees of beliefs based on possibly inconsistent, incomplete, and uncertain evidence. We show that computing degrees of belief with this model is #P-complete in general.
  arXiv  Detail & Related papers  (2023-06-06T09:30:48Z)
• Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence [59.81749318292707]
  We propose a fact verification model named ReRead to retrieve evidence and verify claim. The proposed system is able to achieve significant improvements upon best-reported models under different settings.
  arXiv  Detail & Related papers  (2023-05-02T03:23:14Z)
• Uncertain Evidence in Probabilistic Models and Stochastic Simulators [80.40110074847527]
  We consider the problem of performing Bayesian inference in probabilistic models where observations are accompanied by uncertainty, referred to as uncertain evidence' We explore how to interpret uncertain evidence, and by extension the importance of proper interpretation as it pertains to inference about latent variables. We devise concrete guidelines on how to account for uncertain evidence and we provide new insights, particularly regarding consistency.
  arXiv  Detail & Related papers  (2022-10-21T20:32:59Z)
• Revealing Unfair Models by Mining Interpretable Evidence [50.48264727620845]
  The popularity of machine learning has increased the risk of unfair models getting deployed in high-stake applications. In this paper, we tackle the novel task of revealing unfair models by mining interpretable evidence. Our method finds highly interpretable and solid evidence to effectively reveal the unfairness of trained models.
  arXiv  Detail & Related papers  (2022-07-12T20:03:08Z)
• Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation [19.551623461082617]
  Open-domain question answering answers a question based on evidence retrieved from a large corpus. This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training. We introduce a novel approach (DistDR) that iteratively improves over a weak retriever by alternately finding evidence from the up-to-date model and encouraging the model to learn the most likely evidence.
  arXiv  Detail & Related papers  (2021-10-10T20:01:27Z)
• Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning [26.168211982441875]
  Neural models routinely report state-of-the-art performance across NLP tasks involving reasoning. Our experiments demonstrate that a BERT-based model representative of today's state-of-the-art fails to properly reason on the following counts.
  arXiv  Detail & Related papers  (2021-08-02T01:14:19Z)
• Topic-Aware Evidence Reasoning and Stance-Aware Aggregation for Fact Verification [19.130541561303293]
  We propose a novel topic-aware evidence reasoning and stance-aware aggregation model for fact verification. Tests conducted on two benchmark datasets demonstrate the superiority of the proposed model over several state-of-the-art approaches for fact verification.
  arXiv  Detail & Related papers  (2021-06-02T14:33:12Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.