Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition

• URL: http://arxiv.org/abs/2603.16463v1
• Date: Tue, 17 Mar 2026 12:49:51 GMT
• Title: Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition
• Abstract summary: Open-Vocabulary Multimodal Emotion Recognition (OV-MER) is inherently challenging due to the ambiguity of equivocal multimodal cues.<n>We introduce HyDRA, a Hybrid-evidential Deductive Reasoning Architecture that formalizes inference as a Propose-Verify-Decide protocol.
• Score: 17.899439937763006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-Vocabulary Multimodal Emotion Recognition (OV-MER) is inherently challenging due to the ambiguity of equivocal multimodal cues, which often stem from distinct unobserved situational dynamics. While Multimodal Large Language Models (MLLMs) offer extensive semantic coverage, their performance is often bottlenecked by premature commitment to dominant data priors, resulting in suboptimal heuristics that overlook crucial, complementary affective cues across modalities. We argue that effective affective reasoning requires more than surface-level association; it necessitates reconstructing nuanced emotional states by synthesizing multiple evidence-grounded rationales that reconcile these observations from diverse latent perspectives. We introduce HyDRA, a Hybrid-evidential Deductive Reasoning Architecture that formalizes inference as a Propose-Verify-Decide protocol. To internalize this abductive process, we employ reinforcement learning with hierarchical reward shaping, aligning the reasoning trajectories with final task performance to ensure they best reconcile the observed multimodal cues. Systematic evaluations validate our design choices, with HyDRA consistently outperforming strong baselines--especially in ambiguous or conflicting scenarios--while providing interpretable, diagnostic evidence traces.

Related papers

• Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes [37.743333035442426]
  This paper introduces AD2-Bench, which decomposes reasoning into a structured Chain of Evidence (CoE)<n>Building on this perspective, the authors identify two primary causes of reasoning failure: Spatial Ambiguity and Semantic Uncertainty.<n>They then propose Evidence-grounded Visual Reasoning (EGVOR), which replaces implicit reasoning with the explicit generation of Evidence Atoms.
  arXiv  Detail & Related papers  (2026-08-11T14:23:07Z)
• Explicit Representation Alignment for Multimodal Sentiment Analysis [17.25685888421759]
  Multimodal affective analysis aims to understand human sentiment and emotion by jointly modeling heterogeneous modalities such as text and images.<n>We identify representation misalignment between independently pretrained modality encoders as a key bottleneck for effective multimodal learning.<n>We propose a unified multimodal affective analysis framework that leverages vision-language models (VLMs) to convert visual content into structured textual descriptions.
  arXiv  Detail & Related papers  (2026-06-08T07:43:52Z)
• MM-StanceDet: Retrieval-Augmented Multi-modal Multi-agent Stance Detection [8.10743562429719]
  We propose Retrieval-Augmented Multi-modal Multi-agent Stance Detection (MM-StanceDet)<n>MM-StanceDet is a novel multi-agent framework integrating Retrieval Augmentation for contextual grounding, specialized Multimodal Analysis agents for nuanced interpretation, a Reasoning-Enhanced Debate stage, and Self-Reflection for robust adjudication.<n>Extensive experiments on five datasets demonstrate MM-StanceDet significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-04-30T14:34:18Z)
• V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval [32.5242219186118]
  We propose V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection.<n>V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.
  arXiv  Detail & Related papers  (2026-02-05T18:59:21Z)
• Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning [9.470507126417292]
  We introduce SABER-LLM, a framework designed for robust multimodal reasoning.<n>First, we construct SABER, a large-scale emotion reasoning dataset comprising 600K video clips.<n>Second, we propose the structured evidence decomposition paradigm, which enforces a "perceive-then-reason" separation between evidence extraction and reasoning.
  arXiv  Detail & Related papers  (2026-01-26T10:03:26Z)
• Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models [72.4149653187766]
  We propose a Reasoner-Verifier framework named Adrialversa Reasoning RAG (ARR)<n>The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage.<n> Experiments on multiple benchmarks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2026-01-08T06:57:03Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding [50.014363382140324]
  Modality Importance (MI) is a simple yet effective mechanism for identifying the emotion-dominant modality.<n>MIGR reorganizes reasoning sequences so that explanations begin from the modality most critical to the target emotion.<n>Results show that MIGR substantially improves reasoning reliability, decreasing instances of correct predictions accompanied by emotionally inconsistent explanations.
  arXiv  Detail & Related papers  (2025-12-02T12:29:41Z)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning [49.17801010041155]
  Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio.<n>Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance.<n>We categorize multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined.
  arXiv  Detail & Related papers  (2025-09-28T08:46:11Z)
• Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing [10.66971486730557]
  Multimodal Large Language Models (MLLMs) have shown substantial capabilities in integrating visual and textual information, yet frequently rely on spurious correlations.<n>This paper addresses the critical challenge of superficial correlation bias in MLLMs through a novel causal mediation-based debiasing framework.
  arXiv  Detail & Related papers  (2025-09-18T19:01:11Z)
• Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signals [45.019257216564036]
  This paper investigates extended inductive reasoning in large language models (LLMs)<n>We propose AlignXplore, a model that enables systematic preference inference from behavioral signals in users' interaction histories.<n>We show that AlignXplore achieves substantial improvements over the backbone model by an average of 15.49% on in-domain and out-of-domain benchmarks.
  arXiv  Detail & Related papers  (2025-05-23T16:16:46Z)
• Progressive Multimodal Reasoning via Active Retrieval [64.74746997923967]
  Multi-step multimodal reasoning tasks pose significant challenges for large language models (MLLMs)<n>We propose AR-MCTS, a universal framework designed to progressively improve the reasoning capabilities of MLLMs.<n>We show that AR-MCTS can optimize sampling diversity and accuracy, yielding reliable multimodal reasoning.
  arXiv  Detail & Related papers  (2024-12-19T13:25:39Z)
• MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition [118.73025093045652]
  We propose a pre-training model textbfMEmoBERT for multimodal emotion recognition. Unlike the conventional "pre-train, finetune" paradigm, we propose a prompt-based method that reformulates the downstream emotion classification task as a masked text prediction. Our proposed MEmoBERT significantly enhances emotion recognition performance.
  arXiv  Detail & Related papers  (2021-10-27T09:57:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.