The Impact of Element Ordering on LM Agent Performance

• URL: http://arxiv.org/abs/2409.12089v3
• Date: Sun, 6 Oct 2024 21:25:34 GMT
• Title: The Impact of Element Ordering on LM Agent Performance
• Authors: Wayne Chi, Ameet Talwalkar, Chris Donahue,
• Abstract summary: We investigate the impact of various element ordering methods in web and desktop environments. We find that dimensionality reduction provides a viable ordering for pixel-only environments. Our method completes more than two times as many tasks on average relative to the previous state-of-the-art.
• Score: 25.738019870722482
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There has been a surge of interest in language model agents that can navigate virtual environments such as the web or desktop. To navigate such environments, agents benefit from information on the various elements (e.g., buttons, text, or images) present. It remains unclear which element attributes have the greatest impact on agent performance, especially in environments that only provide a graphical representation (i.e., pixels). Here we find that the ordering in which elements are presented to the language model is surprisingly impactful--randomizing element ordering in a webpage degrades agent performance comparably to removing all visible text from an agent's state representation. While a webpage provides a hierarchical ordering of elements, there is no such ordering when parsing elements directly from pixels. Moreover, as tasks become more challenging and models more sophisticated, our experiments suggest that the impact of ordering increases. Finding an effective ordering is non-trivial. We investigate the impact of various element ordering methods in web and desktop environments. We find that dimensionality reduction provides a viable ordering for pixel-only environments. We train a UI element detection model to derive elements from pixels and apply our findings to an agent benchmark--OmniACT--where we only have access to pixels. Our method completes more than two times as many tasks on average relative to the previous state-of-the-art.

Related papers

• IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web [61.96082780724042]
  We have curated and aligned a benchmark of images and corresponding web codes (IW-Bench) We propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. We also design a five-hop multimodal Chain-of-Thought Prompting for better performance.
  arXiv  Detail & Related papers  (2024-09-14T05:38:26Z)
• OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding [112.87441334765693]
  OMG-LLaVA is a new framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model.
  arXiv  Detail & Related papers  (2024-06-27T17:59:01Z)
• Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP [53.18562650350898]
  We introduce a general framework which can identify the roles of various components in ViTs beyond CLIP. We also introduce a novel scoring function to rank components by their importance with respect to specific features. Applying our framework to various ViT variants we gain insights into the roles of different components concerning particular image features.
  arXiv  Detail & Related papers  (2024-06-03T17:58:43Z)
• Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Network [25.511804582983977]
  Panoramic Narrative Grounding (PNG) aims to segment visual objects in images based on dense narrative captions. We propose a novel learning framework called Deformable Attention Refined Matching Network (DRMN) DRMN iteratively re-encodes pixels with the deformable attention network after updating the feature representation of the top-$k$ most similar pixels.
  arXiv  Detail & Related papers  (2023-10-25T13:12:39Z)
• MvP: Multi-view Prompting Improves Aspect Sentiment Tuple Prediction [14.177875807409434]
  We propose Multi-view Prompting (MvP) that aggregates sentiment elements generated in different orders. MvP can naturally model multi-view and multi-task as permutations and combinations of elements. Extensive experiments show that MvP significantly advances the state-of-the-art performance on 10 datasets of 4 benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-22T01:32:50Z)
• DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation [83.51882381294357]
  DisPositioNet is a model that learns a disentangled representation for each object for the task of image manipulation using scene graphs. Our framework enables the disentanglement of the variational latent embeddings as well as the feature representation in the graph.
  arXiv  Detail & Related papers  (2022-11-10T11:47:37Z)
• Multi-modal Transformers Excel at Class-agnostic Object Detection [105.10403103027306]
  We argue that existing methods lack a top-down supervision signal governed by human-understandable semantics. We develop an efficient and flexible MViT architecture using multi-scale feature processing and deformable self-attention. We show the significance of MViT proposals in a diverse range of applications.
  arXiv  Detail & Related papers  (2021-11-22T18:59:29Z)
• Unsupervised Part Discovery from Contrastive Reconstruction [90.88501867321573]
  The goal of self-supervised visual representation learning is to learn strong, transferable image representations. We propose an unsupervised approach to object part discovery and segmentation. Our method yields semantic parts consistent across fine-grained but visually distinct categories.
  arXiv  Detail & Related papers  (2021-11-11T17:59:42Z)
• Multimodal Icon Annotation For Mobile Applications [11.342641993269693]
  We propose a novel deep learning based multi-modal approach that combines the benefits of both pixel and view hierarchy features. In order to demonstrate the utility provided, we create a high quality UI dataset by manually annotating the most commonly used 29 icons in Rico.
  arXiv  Detail & Related papers  (2021-07-09T13:57:37Z)
• Realizing Pixel-Level Semantic Learning in Complex Driving Scenes based on Only One Annotated Pixel per Class [17.481116352112682]
  We propose a new semantic segmentation task under complex driving scenes based on weakly supervised condition. A three step process is built for pseudo labels generation, which progressively implement optimal feature representation for each category. Experiments on Cityscapes dataset demonstrate that the proposed method provides a feasible way to solve weakly supervised semantic segmentation task.
  arXiv  Detail & Related papers  (2020-03-10T12:57:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.