How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

• URL: http://arxiv.org/abs/2601.21961v1
• Date: Thu, 29 Jan 2026 16:40:15 GMT
• Title: How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors
• Abstract summary: We introduce VAF, a pipeline for quantifying how webpage Visual Attribute Factors influence web-agent decision-making.<n>VAF consists of three stages: (i) variant generation, (ii) browsing interaction, and (iii) validating through both click action and reasoning from agents.<n>By quantitatively measuring the decision-making difference between the original and variant, we identify which visual attributes influence agents' behavior most.
• Score: 12.445105236386008
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Web agents have demonstrated strong performance on a wide range of web-based tasks. However, existing research on the effect of environmental variation has mostly focused on robustness to adversarial attacks, with less attention to agents' preferences in benign scenarios. Although early studies have examined how textual attributes influence agent behavior, a systematic understanding of how visual attributes shape agent decision-making remains limited. To address this, we introduce VAF, a controlled evaluation pipeline for quantifying how webpage Visual Attribute Factors influence web-agent decision-making. Specifically, VAF consists of three stages: (i) variant generation, which ensures the variants share identical semantics as the original item while only differ in visual attributes; (ii) browsing interaction, where agents navigate the page via scrolling and clicking the interested item, mirroring how human users browse online; (iii) validating through both click action and reasoning from agents, which we use the Target Click Rate and Target Mention Rate to jointly evaluate the effect of visual attributes. By quantitatively measuring the decision-making difference between the original and variant, we identify which visual attributes influence agents' behavior most. Extensive experiments, across 8 variant families (48 variants total), 5 real-world websites (including shopping, travel, and news browsing), and 4 representative web agents, show that background color contrast, item size, position, and card clarity have a strong influence on agents' actions, whereas font styling, text color, and item image clarity exhibit minor effects.

Related papers

• Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects [58.98080129016082]
  Feature-Effect Geometry Analysis (FEGA) is an unsupervised framework that removes the same active SAE feature across contexts.<n>We show that a feature can be interpretable and causally relevant without providing a stable direction for steering.
  arXiv  Detail & Related papers  (2026-07-27T16:45:08Z)
• MooD: Perception-Enhanced Efficient Affective Image Editing via Continuous Valence-Arousal Modeling [6.7122126781526745]
  Affective Image Editing (AIE) aims to modify visual content to evoke targeted emotions.<n>We propose MooD, the first framework that directly leverages continuous Valence-Arousal (VA) values as editing instruction.<n>We show MooD achieves superior performance in both affective controllability and visual fidelity while maintaining high efficiency.
  arXiv  Detail & Related papers  (2026-05-04T12:22:43Z)
• A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents [14.1714863221125]
  We present the first systematic study of cross-feature stylistic side effects.<n>We quantify how prompting for one style feature causally affects others via a pairwise Large Language Model evaluation framework.<n>Our results reveal consistent and structured side effects, such as prompting for conciseness significantly reduces perceived expertise.
  arXiv  Detail & Related papers  (2026-01-15T19:18:18Z)
• Automated Detection of Visual Attribute Reliance with a Self-Reflective Agent [58.90049897180927]
  We introduce an automated framework for detecting unintended reliance on visual features in vision models.<n>A self-reflective agent generates and tests hypotheses about visual attributes that a model may rely on.<n>We evaluate our approach on a novel benchmark of 130 models designed to exhibit diverse visual attribute dependencies.
  arXiv  Detail & Related papers  (2025-10-24T17:59:02Z)
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents [58.92444959954643]
  This paper investigates the integration of visual information, specifically webpage screenshots, into behavior simulation via VLMs.<n>We employ SFT for joint action prediction and rationale generation, conditioning on the full interaction context.<n>To further enhance reasoning capabilities, we integrate RL with a hierarchical reward structure, scaled by a difficulty-aware factor.
  arXiv  Detail & Related papers  (2025-10-22T05:07:14Z)
• Cross-Modal Content Optimization for Steering Web Agent Preferences [16.32764997508559]
  We introduce Cross-Modal Preference Steering (CPS) that jointly optimize imperceptible modifications to an item's visual and natural language descriptions.<n>We evaluate CPS on agents powered by state-of-the-art proprietary and open source VLMs including GPT-4.1, Qwen-2.5VL and Pixtral-Large on both movie selection and e-commerce tasks.
  arXiv  Detail & Related papers  (2025-10-04T01:57:20Z)
• RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation [67.38036090822982]
  We propose RoboView-Bias, the first benchmark specifically designed to quantify visual bias in robotic manipulation.<n>We create 2,127 task instances that enable robust measurement of biases induced by individual visual factors and their interactions.<n>Our results highlight that systematic analysis of visual bias is a prerequisite for developing safe and reliable general-purpose embodied agents.
  arXiv  Detail & Related papers  (2025-09-26T13:53:25Z)
• Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks [2.1117030125341385]
  Large language models (LLMs) enable conversational agents (CAs) to express distinctive personalities.<n>This study investigates how personality expression levels and user-agent personality alignment influence perceptions in goal-oriented tasks.
  arXiv  Detail & Related papers  (2025-09-11T21:43:49Z)
• A Quantitative Evaluation of the Expressivity of BMI, Pose and Gender in Body Embeddings for Recognition and Identification [56.10719736365069]
  We extend the notion of expressivity, defined as the mutual information between learned features and specific attributes, to quantify how strongly attributes are encoded.<n>We find that BMI consistently shows the highest expressivity in the final layers, indicating its dominant role in recognition.<n>These findings demonstrate the central role of body attributes in ReID and establish a principled approach for uncovering attribute driven correlations.
  arXiv  Detail & Related papers  (2025-03-09T05:15:54Z)
• The Impact of Element Ordering on LM Agent Performance [25.738019870722482]
  We investigate the impact of various element ordering methods in web and desktop environments. We find that dimensionality reduction provides a viable ordering for pixel-only environments. Our method completes more than two times as many tasks on average relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2024-09-18T16:04:10Z)
• The Influences of Color and Shape Features in Visual Contrastive Learning [0.0]
  This paper investigates the influences of individual image features (e.g., color and shape) to model performance remain ambiguous. Experimental results show that compared with supervised representations, contrastive representations tend to cluster with objects of similar color.
  arXiv  Detail & Related papers  (2023-01-29T15:10:14Z)
• TransFA: Transformer-based Representation for Face Attribute Evaluation [87.09529826340304]
  We propose a novel textbftransformer-based representation for textbfattribute evaluation method (textbfTransFA) The proposed TransFA achieves superior performances compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-12T10:58:06Z)
• Deep Collaborative Multi-Modal Learning for Unsupervised Kinship Estimation [53.62256887837659]
  Kinship verification is a long-standing research challenge in computer vision. We propose a novel deep collaborative multi-modal learning (DCML) to integrate the underlying information presented in facial properties. Our DCML method is always superior to some state-of-the-art kinship verification methods.
  arXiv  Detail & Related papers  (2021-09-07T01:34:51Z)
• AttributeNet: Attribute Enhanced Vehicle Re-Identification [70.89289512099242]
  We introduce AttributeNet (ANet) that jointly extracts identity-relevant features and attribute features. We enable the interaction by distilling the ReID-helpful attribute feature and adding it into the general ReID feature to increase the discrimination power. We validate the effectiveness of our framework on three challenging datasets.
  arXiv  Detail & Related papers  (2021-02-07T19:51:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.