Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective

• URL: http://arxiv.org/abs/2312.11214v1
• Date: Mon, 18 Dec 2023 14:04:51 GMT
• Title: Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
• Abstract summary: Generative Adversarial Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with policy (ST-GAIL)
• Score: 22.267502760766398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.

Related papers

• Less is More: Early Stopping Rollout for On-Policy Distillation [39.392596318514244]
  Early Stopping Rollout (ESR) is a simple yet effective distillation strategy that simply restricts the rollout generation to the first response tokens.<n>We show that ESR both surpasses the full rollout OPD performance across model size, family, tasks and training regime.
  arXiv  Detail & Related papers  (2026-05-26T13:49:37Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler [54.10960908347221]
  We model latent thought exploration as conditional sampling from learnable densities and instantiate this idea as a Gaussian Thought Sampler (GTS)<n>GTS predicts context-dependent perturbation distributions over continuous reasoning states and is trained with GRPO-style policy optimization while keeping the backbone frozen.
  arXiv  Detail & Related papers  (2026-02-15T09:57:47Z)
• Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training [76.12556589212666]
  We show that curriculum post-training avoids the exponential complexity bottleneck.<n>Under outcome-only reward signals, reinforcement learning finetuning achieves high accuracy with sample complexity.<n>We establish guarantees for test-time scaling, where curriculum-aware querying reduces both reward oracle calls and sampling cost from exponential to order.
  arXiv  Detail & Related papers  (2025-11-10T18:29:54Z)
• On Discovering Algorithms for Adversarial Imitation Learning [28.812210809286086]
  We present emphDiscovered Adversarial Imitation Learning (DAIL), the first meta-learnt AIL algorithm.<n>We show that DAIL generalises across unseen environments and policy optimization algorithms.<n>We also analyse why DAIL leads to more stable training, offering novel insights into the role of RA functions in the stability of AIL.
  arXiv  Detail & Related papers  (2025-10-01T14:02:05Z)
• Diffusion-Reward Adversarial Imitation Learning [33.81857550294019]
  Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. Generative adversarial imitation learning (GAIL) formulates imitation learning as adversarial learning. We propose Diffusion-Reward Adversarial Imitation Learning (DRAIL), which integrates a diffusion model into GAIL.
  arXiv  Detail & Related papers  (2024-05-25T11:53:23Z)
• C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory [20.38647732528661]
  Generative Adversarial Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Imitation Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. Recent work has shown that control theory can help with the convergence of a GAN's training.
  arXiv  Detail & Related papers  (2024-02-26T07:07:00Z)
• A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories [122.11358440078581]
  offline imitation is useful in real-world scenarios where arbitrary interactions are costly and expert actions are unavailable. We propose Trajectory-Aware Learning from Observations (TAILO) to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available.
  arXiv  Detail & Related papers  (2023-11-02T15:41:09Z)
• FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning [66.56240101249803]
  We study how hardening benign clients can affect the global model (and the malicious clients) We propose a trigger reverse engineering based defense and show that our method can achieve improvement with guarantee robustness. Our results on eight competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FL backdoor attacks.
  arXiv  Detail & Related papers  (2022-10-23T22:24:03Z)
• Model-Aware Contrastive Learning: Towards Escaping the Dilemmas [11.27589489269041]
  Contrastive learning (CL) continuously achieves significant breakthroughs across multiple domains. InfoNCE-based methods suffer from some dilemmas, such as textituniformity-tolerance dilemma (UTD) and textitgradient reduction (UTD) We present a Model-Aware Contrastive Learning (MACL) strategy, whose temperature is adaptive to the magnitude of alignment that reflects the basic confidence of the instance discrimination task.
  arXiv  Detail & Related papers  (2022-07-16T08:21:55Z)
• SS-MAIL: Self-Supervised Multi-Agent Imitation Learning [18.283839252425803]
  Two families of algorithms - Behavioral Cloning (BC) and Adversarial Imitation Learning (AIL) BC approaches suffer from compounding errors, as they ignore the sequential decision-making nature of the trajectory generation problem. AIL methods are plagued with instability in their training dynamics. We introduce a novel self-supervised loss that encourages the discriminator to approximate a richer reward function.
  arXiv  Detail & Related papers  (2021-10-18T01:17:50Z)
• On Proximal Policy Optimization's Heavy-tailed Gradients [150.08522793940708]
  We study the heavy-tailed nature of the gradients of the Proximal Policy Optimization surrogate reward function. In this paper, we study the effects of the standard PPO clippings, demonstrating that these tricks primarily serve to offset heavy-tailedness in gradients. We propose incorporating GMOM, a high-dimensional robust estimator, into PPO as a substitute for three clipping tricks.
  arXiv  Detail & Related papers  (2021-02-20T05:51:28Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence [56.40794592158596]
  We study generative adversarial imitation learning (GAIL) under general MDP and for nonlinear reward function classes. This is the first systematic theoretical study of GAIL for global convergence.
  arXiv  Detail & Related papers  (2020-06-24T06:24:37Z)
• On Computation and Generalization of Generative Adversarial Imitation Learning [134.17122587138897]
  Generative Adversarial Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies. This paper investigates the theoretical properties of GAIL.
  arXiv  Detail & Related papers  (2020-01-09T00:40:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.