Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation

• URL: http://arxiv.org/abs/2012.04687v1
• Date: Wed, 25 Nov 2020 15:00:36 GMT
• Title: Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation
• Authors: Thibault Cordier, Tanguy Urvoy, Lina M. Rojas-Barahona, Fabrice Lef\`evre
• Abstract summary: A learning dialogue agent can infer its behaviour from human-to-human or human-machine conversations. One solution to speedup the learning process is to guide the agent's exploration with the help of an expert. We present several imitation learning strategies for dialogue policy where the guiding expert is a near-optimal handcrafted policy.
• Score: 0.716879432974126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A learning dialogue agent can infer its behaviour from interactions with the users. These interactions can be taken from either human-to-human or human-machine conversations. However, human interactions are scarce and costly, making learning from few interactions essential. One solution to speedup the learning process is to guide the agent's exploration with the help of an expert. We present in this paper several imitation learning strategies for dialogue policy where the guiding expert is a near-optimal handcrafted policy. We incorporate these strategies with state-of-the-art reinforcement learning methods based on Q-learning and actor-critic. We notably propose a randomised exploration policy which allows for a seamless hybridisation of the learned policy and the expert. Our experiments show that our hybridisation strategy outperforms several baselines, and that it can accelerate the learning when facing real humans.

Related papers

• AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents [58.807802111818994]
  We propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. An important feature of our method is the use of image-based rewards for the high-level policy, which allows the agent to learn interactions with objects without manual reward engineering.
  arXiv  Detail & Related papers  (2024-03-19T15:41:39Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations [70.7884839812069]
  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue.
  arXiv  Detail & Related papers  (2023-11-09T18:45:16Z)
• MEGA-DAgger: Imitation Learning with Multiple Imperfect Experts [7.4506213369860195]
  MEGA-DAgger is a new DAgger variant that is suitable for interactive learning with multiple imperfect experts. We demonstrate that policy learned using MEGA-DAgger can outperform both experts and policies learned using the state-of-the-art interactive imitation learning algorithms.
  arXiv  Detail & Related papers  (2023-03-01T16:40:54Z)
• Few-Shot Structured Policy Learning for Multi-Domain and Multi-Task Dialogues [0.716879432974126]
  Graph neural networks (GNNs) show a remarkable superiority by reaching a success rate above 80% with only 50 dialogues, when learning from simulated experts. We suggest to concentrate future research efforts on bridging the gap between human data, simulators and automatic evaluators in dialogue frameworks.
  arXiv  Detail & Related papers  (2023-02-22T08:18:49Z)
• Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration [51.268988527778276]
  We present a method for learning a human-robot collaboration policy from human-human collaboration demonstrations. Our method co-optimizes a human policy and a robot policy in an interactive learning process.
  arXiv  Detail & Related papers  (2021-08-13T03:14:43Z)
• Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems [58.724629408229205]
  We demonstrate how traditional supervised learning and a simulator-free adversarial learning method can be used to achieve performance comparable to state-of-the-art RL-based methods. Our main goal is not to beat reinforcement learning with supervised learning, but to demonstrate the value of rethinking the role of reinforcement learning and supervised learning in optimizing task-oriented dialogue systems.
  arXiv  Detail & Related papers  (2020-09-21T12:04:18Z)
• Deep Reinforcement Learning with Interactive Feedback in a Human-Robot Environment [1.2998475032187096]
  We propose a deep reinforcement learning approach with interactive feedback to learn a domestic task in a human-robot scenario. We compare three different learning methods using a simulated robotic arm for the task of organizing different objects. The obtained results show that a learner agent, using either agent-IDeepRL or human-IDeepRL, completes the given task earlier and has fewer mistakes compared to the autonomous DeepRL approach.
  arXiv  Detail & Related papers  (2020-07-07T11:55:27Z)
• Guided Dialog Policy Learning without Adversarial Learning in the Loop [103.20723982440788]
  A number of adversarial learning methods have been proposed to learn the reward function together with the dialogue policy. We propose to decompose the adversarial training into two steps. First, we train the discriminator with an auxiliary dialogue generator and then incorporate a derived reward model into a common RL method to guide the dialogue policy learning.
  arXiv  Detail & Related papers  (2020-04-07T11:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.