StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

• URL: http://arxiv.org/abs/2606.00148v1
• Date: Fri, 29 May 2026 03:20:05 GMT
• Title: StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning
• Abstract summary: Multimodal large language models (MLLMs) often know the rule but pick the wrong answer on visual reasoning tasks.<n>Existing benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal.<n>We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions.
• Score: 13.74158437063828
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual reasoning breaks down, identifying rule-to-instance binding as a concrete next target for vision-grounded reasoning.

Related papers

• Guide, Not Bind: Why Defeasible Priors Fail in Augmented Lagrangian Causal Discovery [0.0]
  Differentiable causal discovery methods encode expert priors as forbidden-edge constraints enforced by an Augmented Lagrangian penalty.<n>We show this design, which we call emphguide, not bind, fails for two independent, precisely characterized reasons.
  arXiv  Detail & Related papers  (2026-09-03T06:53:34Z)
• V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning [59.47823383593251]
  Vision-sampling models can produce fluent answers that are insufficiently grounded in the visual evidence.<n>We introduce Visuals-Based Reinforcement Learning, which decomposes reference responses into atomic propositions.<n>We train our model based on the same SFT checkpoint using component-wise, prefix-localized rubric credit.
  arXiv  Detail & Related papers  (2026-08-26T09:40:46Z)
• Grading the Graders: Verification Autonomy Levels (L0-L5) for LLM Reasoning [0.0]
  We propose Verification Autonomy Levels (VAL), a meta-standard that classifies any verification scheme along a single axis.<n>VAL ranges from L0 (LLM self-declaration; no deterministic anchor) through L2 (objective ground truth; correctness only) to L3 (decidable systems with single-property or domain-level completeness)<n>Central to VAL is the completeness blind spot: substitution- and sampling-based verifiers can confirm that proposed candidates hold, but cannot prove that no candidate was missed.
  arXiv  Detail & Related papers  (2026-08-19T15:10:00Z)
• Knowledge before Reasoning: EC-Reason-Bench, a Training-Free Diagnostic Benchmark for LLM Enzyme Classification [36.44850004100564]
  We propose EC-Reason-Bench, a training-free, diagnostic evaluation protocol built to answer two questions.<n>First, external knowledge is decisive and must precede reasoning.<n>Second, in closed-book settings, whether cascading and chain-of-thought help or hurt depends on a model's tendency to abstain.
  arXiv  Detail & Related papers  (2026-07-29T02:16:09Z)
• Interaction Scaling: Grounding the Third Axis of Test-Time Compute [0.33451037881913753]
  We argue that a single variable governs this third axis, grounding, and that it must hold on both sides of the loop.<n>On hard coding tasks at a fixed token budget, reasoning-only and best-of-N sampling both plateau.<n>Our proposer-reviewer harness reaches a perfect 100% pass rate with no run-to-run variance.
  arXiv  Detail & Related papers  (2026-07-13T14:22:06Z)
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games [69.57330692969543]
  We introduce RNG-Bench, a benchmark suite designed to isolate a base model's ability to reconstruct past observations.<n>RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map.<n>The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs.
  arXiv  Detail & Related papers  (2026-06-17T17:59:34Z)
• VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning? [79.44137632338062]
  When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence?<n>We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question pairs across cardinality, attribute, spatial, and logic tasks.<n>We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images.
  arXiv  Detail & Related papers  (2026-06-05T22:06:07Z)
• DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations [0.0]
  Existing hallucination classify errors by what is wrong with the output -- memorised misconceptions, reasoning failures, fluent fabrications.<n>These are useful for diagnosis but cannot answer a different question: which uncertainty scorer would have caught this error?<n>We propose a complementary taxonomy that classifies errors by their detectability signature -- the signal a scorer family would read.
  arXiv  Detail & Related papers  (2026-06-01T14:11:11Z)
• Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)? [72.2500547961037]
  spatial reasoning is a fundamental capability for vision-language models (VLMs) deployed in real-world environments.<n>We introduce two types of observation challenges: (1) occlusion, which hides target information, and (2) perspective ambiguity, which produces misleading visual cues.<n>For each configuration, we design spatial questions that are answerable under clean observations but require abstention under the introduced challenges.
  arXiv  Detail & Related papers  (2026-05-28T20:44:47Z)
• Pramana: A Protocol-Layer Treatment of Claim Verification in Autonomous Agent Networks [0.0]
  Production verification today splits into two halves: Probabilistic verdict patterns (self-consistency voting, reviewer LLM ensembles) produce judgments, not artifacts.<n>Pramana defines the missing wire format. Every consequential agent output is wrapped in a typed ClaimAttestation with one of four variants.<n>Pramana exhaustively verified under TLC across three symmetry-reduced models: 38,563 distinct reachable states, zero invariant violations.
  arXiv  Detail & Related papers  (2026-05-19T17:00:33Z)
• Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems [52.83669998269706]
  Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
  arXiv  Detail & Related papers  (2026-04-16T09:23:22Z)
• Pando: Do Interpretability Methods Work When Models Won't Explain Themselves? [53.07826484214082]
  We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis.<n>Pando predicts held-out model decisions from $10$ labeled query-response pairs.<n>When explanations are faithful, black-box elicitation matches or exceeds all white-box methods.
  arXiv  Detail & Related papers  (2026-04-13T06:42:24Z)
• Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation [3.23600523782706]
  We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning.<n>We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which penalizes disordered reasoning chains.<n>CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
  arXiv  Detail & Related papers  (2026-03-13T15:48:15Z)
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning [0.6138671548064355]
  Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning.<n>We introduce C2-Faith, a benchmark that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and coverage.<n>We evaluate three frontier judges under three tasks: binary causal detection, causal step localization, and coverage scoring.
  arXiv  Detail & Related papers  (2026-03-05T13:36:47Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings [91.3041144166326]
  We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:39:40Z)
• Dual-Stage Reweighted MoE for Long-Tailed Egocentric Mistake Detection [85.0189917888094]
  We propose a Dual-Stage Reweighted Mixture-of-Experts (DR-MoE) framework to handle the challenges posed by subtle and infrequent mistakes.<n>The proposed method achieves strong performance, particularly in identifying rare and ambiguous mistake instances.
  arXiv  Detail & Related papers  (2025-09-16T12:00:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.