C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models

• URL: http://arxiv.org/abs/2608.05381v1
• Date: Wed, 05 Aug 2026 20:04:05 GMT
• Title: C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
• Abstract summary: We introduce C$3$PO, a benchmark of 3,404 samples spanning video, audio, image, and text.<n>We evaluate two abilities: information composition (fusing dispersed evidence) and counterfactual conflict.<n>Built through a fully automatic pipeline using 25 logically grounded templates, C$3$PO reveals that while humans achieve 88.64% accuracy, the best model reaches only 73.17%.
• Score: 11.992819604307998
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current Multimodal Large Language Models (MLLMs) can process diverse sensory inputs, yet their reasoning remains heavily biased toward a dominant modality, resulting in brittle cross-modal reasoning. We introduce C$^3$PO, a benchmark of 3,404 samples spanning video, audio, image, and text, evaluating two abilities: information composition (fusing dispersed evidence) and counterfactual conflict (resolving deliberate contradictions). C$^3$PO's paired IC/CC structure and four-tier design enable targeted diagnosis of when and why cross-modal reasoning fails. Built through a fully automatic pipeline using 25 logically grounded templates, C$^3$PO reveals that while humans achieve 88.64% accuracy, the best model (Gemini-3.1-Pro) reaches only 73.17%, with open-source models collapsing under conflict. Through attention probes, we find 86-95% of failures stem from modality dominance: models commit to one modality while ignoring contradictory evidence, concentrating 87-95% of attention on text. Mid-layer attention entropy predicts correctness-sustained exploration succeeds, premature collapse fails. The 56-point accuracy gap between equally complex templates reveals that performance depends on modalities' structural roles in conflict resolution, not combinations. These findings show multimodal perception does not guarantee robust reasoning; architectures must enable sustained cross-modal attention to avoid premature

Related papers

• Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment [53.50824432699408]
  Traits Run Deeper is a novel personality assessment framework.<n>It consists of three components: the Multimodal Foundation Representation (MFR), the Trait-Specific Modality Fusion (TSMF) and the Distribution-Calibrated Personality Regression (DCPR)
  arXiv  Detail & Related papers  (2026-06-09T06:38:36Z)
• FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses [0.0]
  FailureScope is a behavioral-diagnosis method that clusters evaluation probes by their cross-model pass/fail patterns.<n>We show it yields stable, interpretable failure across three regimes usually studied separately: single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks.
  arXiv  Detail & Related papers  (2026-06-03T01:28:00Z)
• It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers [0.0]
  A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability.<n>We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers.<n>We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures.
  arXiv  Detail & Related papers  (2026-05-26T09:08:41Z)
• Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning [56.48520300004217]
  We present Seirnes, a self-play RL framework that transforms contextual interference into an internal training signal.<n>A single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots.<n>By pitting these competing objectives against each other, Seirnes compels the model to move beyond superficial pattern matching.
  arXiv  Detail & Related papers  (2026-05-12T06:58:35Z)
• Semantic Invariance in Agentic AI [2.7821684674538347]
  Large Language Models increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems.<n> deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations.<n>Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension.
  arXiv  Detail & Related papers  (2026-03-13T17:08:44Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models [0.0]
  We present a controlled study of multi-hop contextual reasoning in large language models.<n>We show that multi-agent systems show the inverse pattern, achieving up to 80% on reasoning tasks where rule-based methods fail.
  arXiv  Detail & Related papers  (2026-01-06T20:18:55Z)
• CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution [20.823419395675412]
  CrossCheck-Bench is a diagnostic benchmark for evaluating contradiction detection in multimodal inputs.<n>We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection.
  arXiv  Detail & Related papers  (2025-11-19T12:17:15Z)
• Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoning [70.56067503630486]
  We argue that sixth-generation (6G) intelligence is not fluent token prediction but calibrated the capacity to imagine and choose.<n>We show that WM-MS3M cuts mean absolute error (MAE) by 1.69% versus MS3M with 32% fewer parameters and similar latency, and achieves 35-80% lower root mean squared error (RMSE) than attention/hybrid baselines with 2.3-4.1x faster inference.
  arXiv  Detail & Related papers  (2025-11-04T17:22:22Z)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning [49.17801010041155]
  Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio.<n>Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance.<n>We categorize multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined.
  arXiv  Detail & Related papers  (2025-09-28T08:46:11Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.