Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs

• URL: http://arxiv.org/abs/2608.12781v2
• Date: Mon, 17 Aug 2026 18:08:51 GMT
• Title: Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
• Abstract summary: We study whether thinking and non-thinking interfaces preserve acceptable final-response behavior.<n>PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning.<n>We develop textbfPatternRM, a response-level reward model, and textbfPatternRL, which introduces pattern-specific penalties during reinforcement learning.
• Score: 46.40763822929566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hybrid-thinking multimodal large language models (MLLMs) allow a single model to alternate between deliberative thinking and latency-efficient non-thinking inference. Although these modes differ in reasoning budget, their delivered responses should satisfy the same user-facing standard. Correctness alone may not characterize this response quality; we therefore evaluate task accuracy and response-pattern failures as complementary outcomes. We study this gap through \textbf{response-pattern alignment}: whether thinking and non-thinking interfaces preserve acceptable final-response behavior. We introduce \textbf{PatternEval}, a failure-enriched diagnostic benchmark comprising 2,415 multimodal prompts spanning visual perception and grounding, structured image understanding, and multimodal knowledge reasoning. PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Response-pattern failures are widespread across models from different providers, with non-thinking inference exhibiting substantially higher failure rates and thereby creating systematic misalignment between thinking and non-thinking interfaces. Motivated by this diagnosis, we develop \textbf{PatternRM}, a response-level reward model, and \textbf{PatternRL}, which introduces pattern-specific penalties during reinforcement learning. Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that incorporating pattern-specific penalties into reinforcement learning can mitigate cross-mode misalignment while incurring a marginal task performance trade-off. Together, PatternEval and PatternRL provide an evaluation-and-training framework for aligning user-visible response patterns across hybrid-thinking interfaces.

Related papers

• Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization [47.64132795594338]
  In factuality-oriented question answering (QA), explicit thinking often improves overall performance by helping the model recover relevant knowledge and refine its answers.<n>We propose MARGO, a reinforcement learning framework that uses non-thinking rollouts as same-model references in advantage estimation.
  arXiv  Detail & Related papers  (2026-07-07T05:34:25Z)
• Robust Fuzzy Multi-view Learning under View Conflict [50.785852124911095]
  existing TMVC methods assume strict alignment across different views during both training and testing phases.<n>To tackle this setting, existing TMVC methods suffer from three critical limitations: underestimated uncertainty, misleading decisions, and overfitting to VC.<n>This paper proposes a novel Robust Fuzzy Multi-View Learning (R-FUML) framework grounded in Fuzzy Set Theory.
  arXiv  Detail & Related papers  (2026-05-23T08:52:34Z)
• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems [52.83669998269706]
  Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
  arXiv  Detail & Related papers  (2026-04-16T09:23:22Z)
• Recursive Think-Answer Process for LLMs and VLMs [54.52289112197118]
  We propose an efficient Recursive Think-Answer Process (R-TAP)<n>R-TAP enables models to engage in iterative reasoning cycles and generate more accurate answers.<n>We show that R-TAP-enhanced models consistently outperform conventional single-pass methods.
  arXiv  Detail & Related papers  (2026-03-02T17:20:10Z)
• Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities? [61.533560295383786]
  Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture.<n>We observe that U-MLLMs fail to maintain semantic equivalence when required to render the same results in the image modality.<n>We introduce VGUBench, a framework to decouple reasoning logic from generation fidelity.
  arXiv  Detail & Related papers  (2026-02-27T06:23:56Z)
• CODE: A Contradiction-Based Deliberation Extension Framework for Overthinking Attacks on Retrieval-Augmented Generation [43.85448261466922]
  We propose an end-to-end attack framework named Contradiction-Based Deliberation Extension (CODE)<n> CODE develops a multi-agent architecture to construct poisoning samples that are injected into the knowledge base.<n>Experiments show that CODE causes a 5.32x-24.72x increase in reasoning token consumption, without degrading task performance.
  arXiv  Detail & Related papers  (2026-01-19T14:52:31Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution [20.823419395675412]
  CrossCheck-Bench is a diagnostic benchmark for evaluating contradiction detection in multimodal inputs.<n>We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection.
  arXiv  Detail & Related papers  (2025-11-19T12:17:15Z)
• Think Twice: Branch-and-Rethink Reasoning Reward Model [32.70732791642558]
  We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling.<n>We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks.<n>By converting all-at-oncescoringintofocused, second-lookreasoning, BR-RMreducesjudgmentdiffusionandimproves sensitivity to subtle yet consequential errors.
  arXiv  Detail & Related papers  (2025-10-27T17:58:07Z)
• On Evaluating Performance of LLM Inference Serving Systems [11.712948114304925]
  We identify recurring anti-patterns across three key dimensions: Baseline Fairness, Evaluation setup, and Metric Design.<n>These anti-patterns are uniquely problematic for Large Language Model (LLM) inference due to its dual-phase nature.<n>We provide a comprehensive checklist derived from our analysis, establishing a framework for recognizing and avoiding these anti-patterns.
  arXiv  Detail & Related papers  (2025-07-11T20:58:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.