A data-driven approach for learning to control computers
- URL: http://arxiv.org/abs/2202.08137v1
- Date: Wed, 16 Feb 2022 15:23:46 GMT
- Title: A data-driven approach for learning to control computers
- Authors: Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton,
Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex
Goldin, Adam Santoro, Timothy Lillicrap
- Abstract summary: We investigate the setting of computer control using keyboard and mouse, with goals specified via natural language.
We achieve state-of-the-art and human-level mean performance across all tasks within the MiniWob++ benchmark.
These results demonstrate the usefulness of a unified human-agent interface when training machines to use computers.
- Score: 8.131261634438912
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: It would be useful for machines to use computers as humans do so that they
can aid us in everyday tasks. This is a setting in which there is also the
potential to leverage large-scale expert demonstrations and human judgements of
interactive behaviour, which are two ingredients that have driven much recent
success in AI. Here we investigate the setting of computer control using
keyboard and mouse, with goals specified via natural language. Instead of
focusing on hand-designed curricula and specialized action spaces, we focus on
developing a scalable method centered on reinforcement learning combined with
behavioural priors informed by actual human-computer interactions. We achieve
state-of-the-art and human-level mean performance across all tasks within the
MiniWob++ benchmark, a challenging suite of computer control problems, and find
strong evidence of cross-task transfer. These results demonstrate the
usefulness of a unified human-agent interface when training machines to use
computers. Altogether our results suggest a formula for achieving competency
beyond MiniWob++ and towards controlling computers, in general, as a human
would.
Related papers
Err
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.