Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs

• URL: http://arxiv.org/abs/2605.13737v1
• Date: Wed, 13 May 2026 16:14:44 GMT
• Title: Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs
• Abstract summary: We document a Representation-Action Gap: hidden states reliably encode premise-perception mismatches.<n> Behaviorally, models fall into two failure modes: under-rejection and over-rejection.<n>As an initial diagnostic intervention, a probe-guided logit adjustment re-injects the encoded mismatch signal into decoding.
• Score: 41.451167369762544
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: When an omnimodal large language model accepts a question whose textual premise contradicts what it actually sees or hears, does the failure lie in perception or in action? Recent omnimodal models are positioned as perception-grounded agents that jointly process video, audio, and text, yet a basic form of grounding remains untested: catching a textual claim that conflicts with the model's own sensory input. We introduce IMAVB, a curated 500-clip benchmark of long-form movies with a 2x2 design crossing target modality (vision, audio) and premise condition (standard, misleading), which lets us measure conflict detection separately from ordinary multimodal comprehension. Across eight open-source omnimodal LLMs and Gemini 3.1 Pro, we document a Representation-Action Gap: hidden states reliably encode premise-perception mismatches even when the same models almost never reject the false claim in their outputs. Behaviorally, models fall into two failure modes: under-rejection, in which they answer misleading questions as if the false premise were true; and over-rejection, in which they reject more often but also reject standard questions, sacrificing ordinary comprehension accuracy. The gap is modality-asymmetric (audio grounding underperforms vision) and prompt-resistant across seven variants. As an initial diagnostic intervention, a probe-guided logit adjustment (PGLA) re-injects the encoded mismatch signal into decoding and consistently improves rejection behavior. Together, these results suggest the bottleneck for omnimodal grounding lies in translation, not perception.

Related papers

• Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos? [22.544360578469156]
  We present Video-Holmes-V2, a novel benchmark for Deep Audio-Visual Coupling.<n>It enforces an Evidence-Based Evaluation, requiring models to justify answers with precise-temporal audio-visual evidence.<n>Our method distills high-value reasoning cues to mitigate long-grained noise.
  arXiv  Detail & Related papers  (2026-09-15T14:30:00Z)
• Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts [49.97842835825998]
  Existing benchmarks conflate two forms of evidence within a single modality: perceptual signals and propositional signals.<n>We introduce Tri-PvP, a benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form.<n>We find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias.
  arXiv  Detail & Related papers  (2026-09-05T10:33:58Z)
• They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It [0.0]
  We show that the failure of a language model to decode the sender's intent is one of readouts on top of a robust representation.<n>A linear probe decodes the sender's intent, whether they want a thing recognized or evaluated, across six models and four families and in the base checkpoints.<n>The behavioral half of the story is established on the recognize/evaluate contrast, where what varies is whether the default output acts on the intent.
  arXiv  Detail & Related papers  (2026-07-03T20:49:37Z)
• Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts [7.071615105094556]
  We show that visual attributes can be linearly decodable from early layers.<n>We show that training-free activation in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups.
  arXiv  Detail & Related papers  (2026-04-10T14:36:07Z)
• What's Left Unsaid? Detecting and Correcting Misleading Omissions in Multimodal News Previews [31.373823254968315]
  Even when factually correct, social-media news previews can induce interpretation drift.<n>This covert harm is harder to detect than explicit misinformation yet remains underexplored.<n>We develop a multi-stage pipeline that disentangles and simulates preview-based versus context-based understanding.
  arXiv  Detail & Related papers  (2026-01-09T06:29:19Z)
• e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings [91.3041144166326]
  We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:39:40Z)
• When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals [0.07777489763207261]
  Current evaluations mostly report global rates such as false rejection or compliance.<n>We introduce "semantic confusion," a failure mode that captures such local inconsistency.<n>We propose three model-agnostic metrics at the token level: Confusion Index, Confusion Rate, and Confusion Depth.
  arXiv  Detail & Related papers  (2025-11-30T19:11:45Z)
• Measuring Epistemic Humility in Multimodal Large Language Models [17.490955813494693]
  We present HumbleBench, a new hallucination benchmark designed to evaluate MLLMs' ability to reject plausible but incorrect answers.<n>We leverage fine-grained scene graph annotations to extract ground-truth entities and relations, and prompt GPT-4-Turbo to generate multiple-choice questions.<n>HumbleBench fills a key gap in current evaluation suites, providing a more realistic measure of MLLM reliability in safety-critical settings.
  arXiv  Detail & Related papers  (2025-09-11T17:54:00Z)
• Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability [10.607081850023286]
  We introduce the Input Scrutiny Ability Evaluation Framework (ISEval), which encompasses seven categories of flawed premises and three evaluation metrics.<n>Most models struggle to actively detect flawed textual premises without guidance.<n>These insights underscore the urgent need to enhance LMMs' proactive verification of input validity.
  arXiv  Detail & Related papers  (2025-08-06T02:13:46Z)
• Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models [65.23999399834638]
  We introduce DeceptionDecoded, a benchmark of 12,000 image-caption pairs grounded in trustworthy reference articles.<n>The dataset captures both misleading and non-misleading cases, spanning manipulations across visual and textual modalities.<n>It supports three intent-centric tasks: misleading intent detection, misleading source attribution, and creator desire inference.
  arXiv  Detail & Related papers  (2025-05-21T13:14:32Z)
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition [53.800937914403654]
  Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. We propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality.
  arXiv  Detail & Related papers  (2024-03-07T06:06:55Z)
• Deconfounded Video Moment Retrieval with Causal Intervention [80.90604360072831]
  We tackle the task of video moment retrieval (VMR), which aims to localize a specific moment in a video according to a textual query. Existing methods primarily model the matching relationship between query and moment by complex cross-modal interactions. We propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction.
  arXiv  Detail & Related papers  (2021-06-03T01:33:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.