Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models

• URL: http://arxiv.org/abs/2608.27135v1
• Date: Thu, 27 Aug 2026 13:46:49 GMT
• Title: Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
• Abstract summary: We introduce a speech-augmented contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries.<n>We find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy.
• Score: 17.521958873911846
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs. speech) and language (English vs. Arabic). We introduce a speech-augmented visually grounded contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented reasoning from complete failure. Evaluating recent multimodal models under text and speech in English and Arabic, we find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with speech amplifying partial failures. We make the benchmark publicly available to the community.

Related papers

• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach [52.92973028971813]
  We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition.<n>The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model.<n>Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality.
  arXiv  Detail & Related papers  (2026-07-16T08:04:23Z)
• DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation [20.845447775728545]
  Large language models increasingly emphunderstand dialectal English, yet still emphproduce only standard, US-leaning English.<n>We introduce textbfDiaLLM, which continually pretrains three open-weight language model families on the International Corpus of English.
  arXiv  Detail & Related papers  (2026-07-08T17:24:27Z)
• Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages [24.57041755792604]
  MIDI is a dataset spanning 3 high-, 3 medium-, and 12 low-resource languages.<n>It provides idioms embedded in both sentence-level and conversational contexts, capturing both literal and figurative readings.<n> Benchmarking state-of-the-art models shows that comprehensions degrade in low-resource languages and that, in all resource tiers, literal interpretations are substantially harder than figurative ones.
  arXiv  Detail & Related papers  (2026-06-01T12:16:14Z)
• Learning the Cue or Learning the Word? Analyzing Generalization in Metaphor Detection for Verbs [54.86138668387175]
  We analyze metaphor detection through RoBERTa, the shared backbone of many state-of-the-art systems, focusing on English verbs.<n>We introduce a controlled lexical hold-out setup where all instances of selected target lemmas are strictly excluded from fine-tuning, and compare predictions on these Held-out lemmas against Exposed lemmas.<n>While the model performs best on Exposed lemmas, it maintains robust performance on Held-out lemmas.
  arXiv  Detail & Related papers  (2026-04-15T10:48:09Z)
• Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not [0.0]
  We test whether large language models integrate world knowledge with syntactic structure in a human-like, structure-sensitive way during ambiguity resolution.<n>We construct ambiguous items that keep the syntactic configuration fixed and ensure both parses remain pragmatically possible.<n>In a speeded forced-choice comprehension experiment, humans show a large, correctly directed plausibility effect.<n>Across models, plausibility-driven shifts are weak, unstable, or reversed.
  arXiv  Detail & Related papers  (2026-04-06T16:27:06Z)
• DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation [111.94720088481614]
  Can multimodal generative models effectively produce content given dialectal textual input?<n>We construct a new large-scale benchmark spanning six common English dialects.<n>We design a general encoder-based mitigation strategy for multimodal generative models.
  arXiv  Detail & Related papers  (2025-10-16T17:56:55Z)
• SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models [60.72029578488467]
  SpeechR is a unified benchmark for evaluating reasoning over speech in large audio-language models.<n>It evaluates models along three key dimensions: factual retrieval, procedural inference, and normative judgment.<n> Evaluations on eleven state-of-the-art LALMs reveal that high transcription accuracy does not translate into strong reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-04T03:28:04Z)
• What if Othello-Playing Language Models Could See? [69.77773423053199]
  We introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images.<n>We evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment.
  arXiv  Detail & Related papers  (2025-07-19T07:47:55Z)
• How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations [17.528100902591056]
  Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing.<n> Speech exhibits larger cross-lingual differences than text.<n>For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.
  arXiv  Detail & Related papers  (2024-11-26T18:29:11Z)
• SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning [44.53966523376327]
  SeaEval is a benchmark for multilingual foundation models. We characterize how these models understand and reason with natural language. We also investigate how well they comprehend cultural practices, nuances, and values.
  arXiv  Detail & Related papers  (2023-09-09T11:42:22Z)
• M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval [56.49878599920353]
  This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval. For non-English image-speech retrieval, we outperform the current state-of-the-art performance by a wide margin both when training separate models for each language, and with a single model which processes speech in all three languages.
  arXiv  Detail & Related papers  (2022-11-02T14:54:45Z)
• Provable Limitations of Acquiring Meaning from Ungrounded Form: What will Future Language Models Understand? [87.20342701232869]
  We investigate the abilities of ungrounded systems to acquire meaning. We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence. We find that assertions enable semantic emulation if all expressions in the language are referentially transparent. However, if the language uses non-transparent patterns like variable binding, we show that emulation can become an uncomputable problem.
  arXiv  Detail & Related papers  (2021-04-22T01:00:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.