Identifying User Goals from UI Trajectories

• URL: http://arxiv.org/abs/2406.14314v2
• Date: Sun, 30 Jun 2024 12:33:48 GMT
• Title: Identifying User Goals from UI Trajectories
• Authors: Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, Ido Dagan,
• Abstract summary: This paper introduces the task of goal identification from observed UI trajectories. We propose a novel evaluation metric to assess whether two task descriptions are paraphrased within a specific UI environment. Using our metric and these datasets, we conducted several experiments comparing the performance of humans and state-of-the-art models.
• Score: 19.492331502146886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous agents that interact with graphical user interfaces (GUIs) hold significant potential for enhancing user experiences. To further improve these experiences, agents need to be personalized and proactive. By effectively comprehending user intentions through their actions and interactions with GUIs, agents will be better positioned to achieve these goals. This paper introduces the task of goal identification from observed UI trajectories, aiming to infer the user's intended task based on their GUI interactions. We propose a novel evaluation metric to assess whether two task descriptions are paraphrases within a specific UI environment. By Leveraging the inverse relation with the UI automation task, we utilized the Android-In-The-Wild and Mind2Web datasets for our experiments. Using our metric and these datasets, we conducted several experiments comparing the performance of humans and state-of-the-art models, specifically GPT-4 and Gemini-1.5 Pro. Our results show that Gemini performs better than GPT but still underperforms compared to humans, indicating significant room for improvement.

Related papers

• TinyClick: Single-Turn Agent for Empowering GUI Automation [0.18846515534317265]
  We present a single-turn agent for graphical user interface (GUI) interaction tasks, using Vision-Language Model Florence-2-Base. The agent's primary task is identifying the screen coordinates of the UI element corresponding to the user's command. It demonstrates strong performance on Screenspot and OmniAct, while maintaining a compact size of 0.27B parameters and minimal latency.
  arXiv  Detail & Related papers  (2024-10-09T12:06:43Z)
• Tell Me What's Next: Textual Foresight for Generic UI Representations [65.10591722192609]
  We propose Textual Foresight, a novel pretraining objective for learning UI screen representations. Textual Foresight generates global text descriptions of future UI states given a current UI and local action taken. We train with our newly constructed mobile app dataset, OpenApp, which results in the first public dataset for app UI representation learning.
  arXiv  Detail & Related papers  (2024-06-12T02:43:19Z)
• Attention Weighted Mixture of Experts with Contrastive Learning for Personalized Ranking in E-commerce [21.7796124109]
  We propose Attention Weighted Mixture of Experts (AW-MoE) with contrastive learning for personalized ranking. AW-MoE has been successfully deployed in the JD e-commerce search engine.
  arXiv  Detail & Related papers  (2023-06-08T07:59:08Z)
• First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization [112.40598205054994]
  We formalize this idea as a completely unsupervised objective for optimizing interfaces. We conduct an observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in a variety of domains.
  arXiv  Detail & Related papers  (2022-05-24T21:57:18Z)
• X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback [83.95599156217945]
  We focus on assistive typing applications in which a user cannot operate a keyboard, but can supply other inputs. Standard methods train a model on a fixed dataset of user inputs, then deploy a static interface that does not learn from its mistakes. We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user.
  arXiv  Detail & Related papers  (2022-03-04T00:07:20Z)
• GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information [33.28541180149195]
  We investigate how multiple sliders with and without feedforward visualizations influence users' control of generative models. We found that more control dimensions (sliders) significantly increase task difficulty and user actions. Visualization alone are not always sufficient for users to understand individual control dimensions.
  arXiv  Detail & Related papers  (2022-02-02T11:25:07Z)
• ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces [12.52699475631247]
  We introduce a new pre-trained UI representation model called ActionBert. Our methodology is designed to leverage visual, linguistic and domain-specific features in user interaction traces to pre-train generic feature representations of UIs and their components. Experiments show that the proposed ActionBert model outperforms multi-modal baselines across all downstream tasks by up to 15.5%.
  arXiv  Detail & Related papers  (2020-12-22T20:49:52Z)
• Assisted Perception: Optimizing Observations to Communicate State [112.40598205054994]
  We aim to help users estimate the state of the world in tasks like robotic teleoperation and navigation with visual impairments. We synthesize new observations that lead to more accurate internal state estimates when processed by the user.
  arXiv  Detail & Related papers  (2020-08-06T19:08:05Z)
• Optimizing Interactive Systems via Data-Driven Objectives [70.3578528542663]
  We propose an approach that infers the objective directly from observed user interactions. These inferences can be made regardless of prior knowledge and across different types of user behavior. We introduce Interactive System (ISO), a novel algorithm that uses these inferred objectives for optimization.
  arXiv  Detail & Related papers  (2020-06-19T20:49:14Z)
• Mining Implicit Entity Preference from User-Item Interaction Data for Knowledge Graph Completion via Adversarial Learning [82.46332224556257]
  We propose a novel adversarial learning approach by leveraging user interaction data for the Knowledge Graph Completion task. Our generator is isolated from user interaction data, and serves to improve the performance of the discriminator. To discover implicit entity preference of users, we design an elaborate collaborative learning algorithms based on graph neural networks.
  arXiv  Detail & Related papers  (2020-03-28T05:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.