Visual Credit Audit for Multimodal Spatial Reasoning
Abstract Overview
This paper presents Visual Credit Audit (VCA), a decision-level evaluation framework for multimodal spatial reasoning that determines if benchmark images genuinely support a model's answers better than text-only or blank controls. VCA separates relative image support from context-specific relation responsiveness using a training-free, label-free support audit. Once ground truth labels are applied, the framework yields dependence-credited correctness (D-CC) to identify models relying on linguistic or dataset priors. The audit is evaluated on four open multimodal large language models using the VSR and GSR-COCO benchmarks, augmented with image permutations, controlled source factorial tests, and human-audited image edits.
Novelty
The primary novelty is a two-axis evaluation framework that explicitly disentangles marginal image-support advantages from relation-consistent visual responses, moving beyond aggregate benchmark accuracy or simple no-image counterfactuals. The framework is distinctive for conducting a label-free support audit on a model's declared decision, which allows for a complete decomposition of accuracy into credited and uncredited correct answers as well as support-advantaged errors.
Results
Across four multimodal large language models and two spatial benchmarks, 12.73% to 26.25% of decisions were found to be correct but uncredited, revealing a nontrivial gap between nominal accuracy and image-supported correctness. A deterministic matched same-split image permutation reduced dependence-credited correctness by 21.25 to 47.80 points across all runs. Controlled evidence-source factorial tests showed that among correct-but-uncredited items, directional response to visual relation reversal ranged from 81.57% to 100.00%, with 32.11% fully changing their binary answers.
Key Points
- Visual Credit Audit (VCA) disentangles marginal image support from relation-specific visual responsiveness, establishing them as distinct evaluative properties in spatial reasoning.
- The audit reveals a substantial mass of correct-but-uncredited model decisions (12.73% to 26.25%), demonstrating that standard accuracy metrics often count decisions as successful without requiring stronger support from the benchmark image.
- Additional analyses using matched image permutation, fixed-pixel relation contrasts, and 108 independently audited edits confirm that VCA captures relation-responsive behaviors missed by baseline null-image controls.