Assisted Perception: Optimizing Observations to Communicate State

• URL: http://arxiv.org/abs/2008.02840v1
• Date: Thu, 6 Aug 2020 19:08:05 GMT
• Title: Assisted Perception: Optimizing Observations to Communicate State
• Authors: Siddharth Reddy, Sergey Levine, Anca D. Dragan
• Abstract summary: We aim to help users estimate the state of the world in tasks like robotic teleoperation and navigation with visual impairments. We synthesize new observations that lead to more accurate internal state estimates when processed by the user.
• Score: 112.40598205054994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We aim to help users estimate the state of the world in tasks like robotic teleoperation and navigation with visual impairments, where users may have systematic biases that lead to suboptimal behavior: they might struggle to process observations from multiple sensors simultaneously, receive delayed observations, or overestimate distances to obstacles. While we cannot directly change the user's internal beliefs or their internal state estimation process, our insight is that we can still assist them by modifying the user's observations. Instead of showing the user their true observations, we synthesize new observations that lead to more accurate internal state estimates when processed by the user. We refer to this method as assistive state estimation (ASE): an automated assistant uses the true observations to infer the state of the world, then generates a modified observation for the user to consume (e.g., through an augmented reality interface), and optimizes the modification to induce the user's new beliefs to match the assistant's current beliefs. We evaluate ASE in a user study with 12 participants who each perform four tasks: two tasks with known user biases -- bandwidth-limited image classification and a driving video game with observation delay -- and two with unknown biases that our method has to learn -- guided 2D navigation and a lunar lander teleoperation video game. A different assistance strategy emerges in each domain, such as quickly revealing informative pixels to speed up image classification, using a dynamics model to undo observation delay in driving, identifying nearby landmarks for navigation, and exaggerating a visual indicator of tilt in the lander game. The results show that ASE substantially improves the task performance of users with bandwidth constraints, observation delay, and other unknown biases.

Related papers

• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• What User Behaviors Make the Differences During the Process of Visual Analytics? [1.5285292154680246]
  This work presents a study on a comprehensive data collection of user behaviors, and our analysis approach with time-series classification methods. Our user study collects user behaviors on a diverse set of visualization tasks with two comparable systems, desktop and immersive visualizations. Our results reveal that user behaviors can be distinguished during the process of visual analytics and there is a potentially strong association between the physical behaviors of users and the visualization tasks they perform.
  arXiv  Detail & Related papers  (2023-11-01T17:45:52Z)
• Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning [82.91837418721182]
  Adaptive interfaces can help users perform sequential decision-making tasks. Recent advances in human-in-the-loop machine learning enable such systems to improve by interacting with users. We propose a reinforcement learning algorithm to train an interface to map raw command signals to actions.
  arXiv  Detail & Related papers  (2023-09-07T16:52:27Z)
• Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation [17.41434948048325]
  We conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system. We discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities.
  arXiv  Detail & Related papers  (2023-05-23T09:24:53Z)
• HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing [25.34222794274071]
  We present HOOV, a wrist-worn sensing method that allows VR users to interact with objects outside their field of view. Based on the signals of a single wrist-worn inertial sensor, HOOV continuously estimates the user's hand position in 3-space. Our novel data-driven method predicts hand positions and trajectories from just the continuous estimation of hand orientation.
  arXiv  Detail & Related papers  (2023-03-13T11:25:32Z)
• First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization [112.40598205054994]
  We formalize this idea as a completely unsupervised objective for optimizing interfaces. We conduct an observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in a variety of domains.
  arXiv  Detail & Related papers  (2022-05-24T21:57:18Z)
• Adversarial Imitation Learning from Video using a State Observer [50.45370139579214]
  We introduce a new algorithm called Visual Generative Adversarial Imitation from Observation using a State Observer VGAIfO-SO. At its core, VGAIfO-SO seeks to address sample inefficiency using a novel, self-supervised state observer. We show experimentally in several continuous control environments that VGAIfO-SO is more sample efficient than other IfO algorithms at learning from video-only demonstrations.
  arXiv  Detail & Related papers  (2022-02-01T06:46:48Z)
• Deep Multi-Task Learning for Joint Localization, Perception, and Prediction [68.50217234419922]
  This paper investigates the issues that arise in state-of-the-art autonomy stacks under localization error. We design a system that jointly performs perception, prediction, and localization. Our architecture is able to reuse computation between both tasks, and is thus able to correct localization errors efficiently.
  arXiv  Detail & Related papers  (2021-01-17T17:20:31Z)
• Data-Driven Distributed State Estimation and Behavior Modeling in Sensor Networks [5.817715558396024]
  We formulate the problem of simultaneous state estimation and behavior learning in a sensor network. We propose a simple yet effective solution by extending the Gaussian process-based Bayes filters (GP-BayesFilters) to an online, distributed setting. The effectiveness of the proposed method is evaluated on tracking objects with unknown movement behaviors using both synthetic data and data collected from a multi-robot platform.
  arXiv  Detail & Related papers  (2020-09-22T21:31:18Z)
• Learning Navigation Costs from Demonstration in Partially Observable Environments [24.457042947946025]
  This paper focuses on inverse reinforcement learning (IRL) to enable safe and efficient autonomous navigation in unknown partially observable environments. We develop a cost function representation composed of two parts: a probabilistic occupancy encoder, with recurrent dependence on the observation sequence, and a cost encoder, defined over the occupancy features. Our model exceeds the accuracy of baseline IRL algorithms in robot navigation tasks, while substantially improving the efficiency of training and test-time inference.
  arXiv  Detail & Related papers  (2020-02-26T17:15:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.