GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures

• URL: http://arxiv.org/abs/2609.13308v1
• Date: Thu, 10 Sep 2026 14:52:26 GMT
• Title: GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures
• Abstract summary: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language models.<n>However, naming the part supplies information that a real system must infer, confounding visual grounding, mechanical reasoning, and category-to-action association.<n>We introduce GroundBench, a diagnostic benchmark that separates these explanations through six branch-and-merge conditions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language models, with no model outperforming a constant baseline until the part was named. However, naming the part supplies information that a real system must infer, confounding visual grounding, mechanical reasoning, and category-to-action association. We introduce GroundBench, a diagnostic benchmark that separates these explanations through six branch-and-merge conditions, each adding a controlled information bundle, and a counterfactual re-ask targeting a real alternate part visible in the same image. Across three OpenAI models and 1,068 predictions, supplying the target region without its identity leaves action accuracy at or below the 0.53 majority baseline (0.26, 0.26, and 0.53), although the models largely reproduce the supplied region. Supplying identity without location instead yields 0.74, 0.68, and 0.68. Every above-baseline gain in this curated set occurs where the supplied part category itself determines the action. A no-vision control leaves GPT-5's scores unchanged or improved, providing evidence consistent with substantial category-to-action association. GPT-4o mini declines on one condition, so this interpretation is not universal. Adding joint type and motion axis does not improve accuracy across six model-stratum comparisons. On 74 counterfactual pairs from 32 objects, GPT-5 achieves 0.86 pair-weighted compliance with a 0.07 shortcut rate but fails all observed push-to-lift-vertical cases. GroundBench identifies which supplied information changes affordance behavior and tests whether apparently grounded performance can be reproduced through textual shortcuts.

Related papers

• UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation [0.0]
  UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol.<n>We show the disagreement is substantially about metrics, not models.
  arXiv  Detail & Related papers  (2026-08-02T00:51:44Z)
• QuantWAMs: Calibrating at the Right Granularity for World Action Models [34.540026172139406]
  World Action Models jointly predict future observations and actions, but their iterative denoising and closed-loop execution make deployment costly.<n>We present QuantWAMs, a PTQ framework that aligns quantization decisions with the calibration context defined by model structure, rollout distribution, and task objective.<n>We evaluate QuantWAMs on Fast-WAM and LingBot-VA across RoboTwin 2.0, LIBERO, and real-robot manipulation with an AgiBot G2.
  arXiv  Detail & Related papers  (2026-07-30T15:54:29Z)
• A Controlled Candidate-Set Benchmark for Offline Satellite-Security Plan Decomposition [0.0]
  Some security-development settings require local models that map an objective to an ordered, checkable plan.<n>We present a low-rank decomposition adapter and release a case-disjoint corpus with 24 authored decompositions and 83 derived next-step examples across 24 satellite-security cases.
  arXiv  Detail & Related papers  (2026-07-29T01:18:47Z)
• WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models [37.59051701511746]
  We introduce WorldRoamBench, an open-world benchmark for long-horizon stability across four dimensions.<n>The benchmark comprises 600+ test cases across Nature, Urban, and Indoor scenes in first/third-person views with WASD 10-60s continuous interaction.
  arXiv  Detail & Related papers  (2026-06-30T13:51:44Z)
• ACCORD: Action-Conditioned Contextual Grounding for Language Agents [69.73525608707764]
  ACCORD (Action-Conditioned Contextual Grounding) is an agent framework for adaptive grounding.<n>We show that current agents often fail to do so. They act from assumed rather than observed specifics, overlook information they could have gathered, and fail to incorporate evidence that has already been returned.
  arXiv  Detail & Related papers  (2026-06-15T09:05:55Z)
• Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark [4.626261940793026]
  We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR.<n>Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline.
  arXiv  Detail & Related papers  (2026-05-28T05:49:36Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation [0.05597620745943382]
  Zero-shot vision-language models (VLMs) offer a promptable alternative to task-specific training for gross tumor volume delineation.<n>We study alignment directions in VoxTell on a held-out internal NSCLC tumor dataset.<n> Alignment analyses revealed that anatomical location is the dominant driver of VoxTell's spatial attention.
  arXiv  Detail & Related papers  (2026-05-02T05:49:32Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits [31.98028879922584]
  We introduce SummExecEdit, a novel pipeline and benchmark to assess models on their ability to both detect factual errors and provide accurate explanations.<n>The top-performing model, Claude3-Opus, achieves a joint detection and explanation score of only 0.49 in our benchmark.<n>We identify four primary types of explanation errors, with 45.4% of them involving a focus on completely unrelated parts of the summary.
  arXiv  Detail & Related papers  (2024-12-17T23:26:44Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.