When Contextual Inference Fails: Cancelability in Interactive Instruction Following

• URL: http://arxiv.org/abs/2603.19997v1
• Date: Fri, 20 Mar 2026 14:46:59 GMT
• Title: When Contextual Inference Fails: Cancelability in Interactive Instruction Following
• Abstract summary: We introduce Build What I Mean, an interactive benchmark for contextual meaning construction.<n>In BWIM, models must resolve ambiguity by either performing a contextual inference or requesting clarification at a small communication cost.<n>We observe suboptimal strategies, such as partner-blind over-clarification and question-averse guessing under uncertainty.
• Score: 51.2195840589474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the separation of literal interpretation from contextual inference in a collaborative block-building task where a builder must resolve underspecified instructions using contextual inferences. Building on an existing two-speaker psycholinguistic paradigm -- which contrasts a pragmatically cooperative speaker with one who is only literally reliable -- we introduce Build What I Mean (BWIM), an interactive benchmark for contextual meaning construction. In BWIM, models must resolve ambiguity by either performing a contextual inference or requesting clarification at a small communication cost. Evaluating several state-of-the-art LLMs, we find a dissociation between judgment and action: while models detect speaker unreliability in explicit confidence ratings, they fail to exploit this information to guide efficient clarification behavior. Instead, we observe suboptimal strategies, such as partner-blind over-clarification and question-averse guessing under uncertainty.

Related papers

• Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference [18.334093251358833]
  Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding.<n>Existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues.<n>We introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents.
  arXiv  Detail & Related papers  (2026-09-09T02:40:02Z)
• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions [52.45054413627452]
  TPI-Train is a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling.<n>TPI-Bench is a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy.
  arXiv  Detail & Related papers  (2026-04-19T10:03:42Z)
• The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning [76.96796481909581]
  Internal cognitive processing is instrumental in formulating high-quality responses.<n>We propose a novel method named FLAIR that conducts latent thinking simultaneously with speech perception.<n>Our approach aligns seamlessly with spoken dialogue systems.
  arXiv  Detail & Related papers  (2026-03-18T15:30:29Z)
• Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation [60.63465682731118]
  The performance of egocentric AI agents is fundamentally limited by multimodal intent ambiguity.<n>We introduce the Plug-and-Play Clarifier, a zero-shot and modular framework that decomposes the problem into discrete, solvable sub-tasks.<n>Our framework improves the intent clarification performance of small language models by approximately 30%, making them competitive with significantly larger counterparts.
  arXiv  Detail & Related papers  (2025-11-12T04:28:14Z)
• Improving Dialogue Discourse Parsing through Discourse-aware Utterance Clarification [14.879100851573998]
  We propose a discourse-aware Clarification Module (DCM) to enhance the performance of the dialogue discourse.<n>DCM employs two distinct reasoning processes: clarification type reasoning and discourse goal reasoning.<n>CPO enables to assess the contributions of the clarifications from DCM and provide feedback to optimize the DCM.
  arXiv  Detail & Related papers  (2025-06-18T02:47:14Z)
• Explaining Sources of Uncertainty in Automated Fact-Checking [41.236833314783134]
  CLUE (Conflict-and-Agreement-aware Language-model Uncertainty Explanations) is a framework to generate natural language explanations of model uncertainty.<n>It identifies relationships between spans of text that expose claim-evidence or inter-evidence conflicts and agreements that drive the model's predictive uncertainty.<n> CLUE produces explanations that are more faithful to the model's uncertainty and more consistent with fact-checking decisions.
  arXiv  Detail & Related papers  (2025-05-23T13:06:43Z)
• Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation [5.259846811078731]
  We focus on the concept of ambiguity for clarification, seeking to model and integrate ambiguities in the clarification process.<n>We name this new prompting scheme Ambiguity Type-Chain of Thought (AT-CoT)
  arXiv  Detail & Related papers  (2025-04-16T14:21:02Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning [89.92601337474954]
  Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations. We introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding.
  arXiv  Detail & Related papers  (2023-06-15T10:41:23Z)
• How to Enhance Causal Discrimination of Utterances: A Case on Affective Reasoning [22.11437627661179]
  We propose the incorporation of textiti.i.i.d. noise terms into the conversation process, thereby constructing a structural causal model (SCM) To facilitate the implementation of deep learning, we introduce the cogn frameworks to handle unstructured conversation data, and employ an autoencoder architecture to regard the unobservable noise as learnable "implicit causes"
  arXiv  Detail & Related papers  (2023-05-04T07:45:49Z)
• I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling [104.09033240889106]
  We introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We then compare a structured utterance-based approach of using pre-trained Transformer models for contradiction detection with the typical unstructured approach.
  arXiv  Detail & Related papers  (2020-12-24T18:47:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.