Mitigating Negative Side Effects via Environment Shaping
- URL: http://arxiv.org/abs/2102.07017v1
- Date: Sat, 13 Feb 2021 22:15:00 GMT
- Title: Mitigating Negative Side Effects via Environment Shaping
- Authors: Sandhya Saisubramanian and Shlomo Zilberstein
- Abstract summary: Agents operating in unstructured environments often produce negative side effects (NSE)
We present an algorithm to solve this problem and analyze its theoretical properties.
Empirical evaluation of our approach shows that the proposed framework can successfully mitigate NSE, without affecting the agent's ability to complete its assigned task.
- Score: 27.400267388362654
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Agents operating in unstructured environments often produce negative side
effects (NSE), which are difficult to identify at design time. While the agent
can learn to mitigate the side effects from human feedback, such feedback is
often expensive and the rate of learning is sensitive to the agent's state
representation. We examine how humans can assist an agent, beyond providing
feedback, and exploit their broader scope of knowledge to mitigate the impacts
of NSE. We formulate this problem as a human-agent team with decoupled
objectives. The agent optimizes its assigned task, during which its actions may
produce NSE. The human shapes the environment through minor reconfiguration
actions so as to mitigate the impacts of the agent's side effects, without
affecting the agent's ability to complete its assigned task. We present an
algorithm to solve this problem and analyze its theoretical properties. Through
experiments with human subjects, we assess the willingness of users to perform
minor environment modifications to mitigate the impacts of NSE. Empirical
evaluation of our approach shows that the proposed framework can successfully
mitigate NSE, without affecting the agent's ability to complete its assigned
task.
Related papers
- External Model Motivated Agents: Reinforcement Learning for Enhanced Environment Sampling [3.536024441537599]
Unlike reinforcement learning (RL) agents, humans remain capable multitaskers in changing environments.
We propose an agent influence framework for RL agents to improve the adaptation efficiency of external models in changing environments.
Our results show that our method outperforms the baselines in terms of external model adaptation on metrics that measure both efficiency and performance.
arXiv Detail & Related papers (2024-06-28T23:31:22Z) - Learning Actionable Counterfactual Explanations in Large State Spaces [16.30292272064278]
We consider settings where optimal CFEs correspond to solutions of weighted set cover problems.
In this work, we provide a deep-network learning procedure that we show experimentally is able to achieve strong performance.
Our problem can also be viewed as one of learning an optimal policy in a family of large but deterministic Markov Decision Processes.
arXiv Detail & Related papers (2024-04-25T20:49:03Z) - AntEval: Evaluation of Social Interaction Competencies in LLM-Driven
Agents [65.16893197330589]
Large Language Models (LLMs) have demonstrated their ability to replicate human behaviors across a wide range of scenarios.
However, their capability in handling complex, multi-character social interactions has yet to be fully explored.
We introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods.
arXiv Detail & Related papers (2024-01-12T11:18:00Z) - Malicious Agent Detection for Robust Multi-Agent Collaborative Perception [52.261231738242266]
Multi-agent collaborative (MAC) perception is more vulnerable to adversarial attacks than single-agent perception.
We propose Malicious Agent Detection (MADE), a reactive defense specific to MAC perception.
We conduct comprehensive evaluations on a benchmark 3D dataset V2X-sim and a real-road dataset DAIR-V2X.
arXiv Detail & Related papers (2023-10-18T11:36:42Z) - Moving Forward by Moving Backward: Embedding Action Impact over Action
Semantics [57.671493865825255]
We propose to model the impact of actions on-the-fly using latent embeddings.
By combining these latent action embeddings with a novel, transformer-based, policy head, we design an Action Adaptive Policy.
We show that our AAP is highly performant even when faced, at inference-time with missing actions and, previously unseen, perturbed action space.
arXiv Detail & Related papers (2023-04-24T17:35:47Z) - Feedback Effect in User Interaction with Intelligent Assistants: Delayed
Engagement, Adaption and Drop-out [9.205174767678365]
This paper identifies and quantifies the feedback effect, a novel component in IA-user interactions.
We show that unhelpful responses from the IA cause users to delay or reduce subsequent interactions.
As users discover the limitations of the IA's understanding and functional capabilities, they learn to adjust the scope and wording of their requests.
arXiv Detail & Related papers (2023-03-17T21:39:33Z) - Formalizing the Problem of Side Effect Regularization [81.97441214404247]
We propose a formal criterion for side effect regularization via the assistance game framework.
In these games, the agent solves a partially observable Markov decision process.
We show that this POMDP is solved by trading off the proxy reward with the agent's ability to achieve a range of future tasks.
arXiv Detail & Related papers (2022-06-23T16:36:13Z) - Inverse Online Learning: Understanding Non-Stationary and Reactionary
Policies [79.60322329952453]
We show how to develop interpretable representations of how agents make decisions.
By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem.
We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them.
Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
arXiv Detail & Related papers (2022-03-14T17:40:42Z) - Information is Power: Intrinsic Control via Information Capture [110.3143711650806]
We argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model.
This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states.
arXiv Detail & Related papers (2021-12-07T18:50:42Z) - Causal Influence Detection for Improving Efficiency in Reinforcement
Learning [11.371889042789219]
We introduce a measure of situation-dependent causal influence based on conditional mutual information.
We show that it can reliably detect states of influence.
All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.
arXiv Detail & Related papers (2021-06-07T09:21:56Z) - On the Sensory Commutativity of Action Sequences for Embodied Agents [2.320417845168326]
We study perception for embodied agents under the mathematical formalism of group theory.
We introduce the Sensory Commutativity Probability criterion which measures how much an agent's degree of freedom affects the environment.
We empirically illustrate how SCP and the commutative properties of action sequences can be used to learn about objects in the environment.
arXiv Detail & Related papers (2020-02-13T16:58:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.