Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

• URL: http://arxiv.org/abs/2607.04334v1
• Date: Sun, 05 Jul 2026 14:33:29 GMT
• Title: Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
• Abstract summary: We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors.<n>We measure it with paired single-channel interventions over 310 real web, mobile, and desktop probes.<n>Our central metric is the Perception-Fusion Gap, the fraction of probes a model perceives correctly yet resolves toward structure under conflict.
• Score: 5.717981279944186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a DOM or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 310 real web, mobile, and desktop probes. Every probe is scored by deterministic forced choice, with no model-generated item and no model judge. Our central metric is the Perception-Fusion Gap, the fraction of probes a model perceives correctly yet resolves toward structure under conflict. Across five models from three vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and Perception-Fusion Gap is positive for every model; non-text identity, by contrast, stays largely pixel-bound. The substitution is specific to the serialized-text and indexed-action channel, and coordinate-action agents are largely immune. For textual conflicts, a white-box ablation traces the effect to a single copied structural value, and in two live environments the conflict drives wrong actions and real task failure. Visual state reliance therefore gives a measurable diagnostic of whether agent state beliefs are visually grounded, and the errors it exposes propagate to actions.

Related papers

• MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents [74.83535434786145]
  MNIST-PRO is a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints.<n>We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas.
  arXiv  Detail & Related papers  (2026-08-31T16:05:39Z)
• Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer [18.616107275409764]
  Methods that transfer predictions from two-dimensional foundation models into three-dimensional segmentation are commonly grouped by task or representation.<n>We introduce textbfLift, Associate, and Fuse (LAF), a decision-centric framework that represents a transfer system as five operators.<n>LAF defines an explicit contract for the persistent carrier and identifies the first stage at which discarded evidence becomes unrecoverable.
  arXiv  Detail & Related papers  (2026-08-21T01:33:23Z)
• Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning [75.00741348880044]
  Tool-augmented vision-language models increasingly "think with images"<n>Recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little.<n>We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives.
  arXiv  Detail & Related papers  (2026-08-10T14:52:10Z)
• When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering [0.8594140167290097]
  A model's agreement across perturbed inputs is used as a label-free reliability signal.<n>Rerendering beats resampling on both accuracy and reliability.<n>Fine-tuning on the model's own cross-render consensus inverts.
  arXiv  Detail & Related papers  (2026-08-06T07:08:05Z)
• Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources [3.168307501254055]
  We show that vision-language models (VLMs) increasingly read news and web content as images, where the publisher's identity is visually present.<n>We introduce Cue, a cross-model diagnostic that measures which surface source cue overrides an article's content evidence.
  arXiv  Detail & Related papers  (2026-07-03T14:22:05Z)
• Can Machines Really See Objects in Images? A Study Based on Syntactic Distance and Visual Self-Referential Instances [53.65561607407989]
  We ask whether a model can still classify correctly when such local cues provide no stable basis for distinction.<n>Experiments on ResNets and Vision Transformers reveal a consistent phase-transition phenomenon.<n>Results suggest that general intelligence may require creating new language, not reusing an existing one.
  arXiv  Detail & Related papers  (2026-06-28T14:27:40Z)
• Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models [29.66841995436342]
  Modern vision-language models (VLMs) remain unreliable on metric distance queries.<n>We introduce textbfViewDiag, a controlled multi-view evaluation protocol built from Hypersim, ScanNet, and KITTI360.<n>We show that stable predictions may reflect prior-driven collapse rather than evidence-sensitive reasoning.
  arXiv  Detail & Related papers  (2026-06-01T18:06:08Z)
• A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration [0.0]
  Production language-model systems answer a request byspanning it across an invisible orchestration of worker agents.<n>We ask what this does to a class of defect no single worker can see.<n>We vary only the model -- ten systems across five generations from one developer and five providers from distinct alignment paradigms.
  arXiv  Detail & Related papers  (2026-05-25T05:09:48Z)
• Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions [2.9521041993295607]
  Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions.<n>We present a tool-guided inference framework that addresses this failure mode without any model training.<n>We report three empirical observations that we believe warrant additional investigation.
  arXiv  Detail & Related papers  (2026-03-31T08:35:10Z)
• Contextual inference from single objects in Vision-Language models [10.367669666212473]
  How much scene context a single object carries is a well-studied question in human scene perception.<n>How this capacity is organized in vision-language models (VLMs) remains poorly understood.<n>We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects.
  arXiv  Detail & Related papers  (2026-03-20T13:24:15Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• Counterfactual Image Editing [54.21104691749547]
  Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. We formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images. We develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.
  arXiv  Detail & Related papers  (2024-02-07T20:55:39Z)
• ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection [70.11264880907652]
  Recent object (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. We propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and camouflaged zooming in and out. Our framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.
  arXiv  Detail & Related papers  (2023-10-31T06:11:23Z)
• Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label Diffusion [51.11295961195151]
  We exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels. Based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion scheme. Our scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets.
  arXiv  Detail & Related papers  (2022-06-10T05:16:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.