C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

• URL: http://arxiv.org/abs/2402.16349v2
• Date: Tue, 29 Oct 2024 03:39:16 GMT
• Title: C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory
• Abstract summary: Generative Adversarial Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Imitation Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. Recent work has shown that control theory can help with the convergence of a GAN's training.
• Score: 20.38647732528661
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative Adversarial Imitation Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. A major drawback of GAIL is its training instability - it inherits the complex training dynamics of GANs, and the distribution shift introduced by RL. This can cause oscillations during training, harming its sample efficiency and final policy performance. Recent work has shown that control theory can help with the convergence of a GAN's training. This paper extends this line of work, conducting a control-theoretic analysis of GAIL and deriving a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting. Based on this, we propose a practical algorithm 'Controlled-GAIL' (C-GAIL). On MuJoCo tasks, our controlled variant is able to speed up the rate of convergence, reduce the range of oscillation and match the expert's distribution more closely both for vanilla GAIL and GAIL-DAC.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training [5.633817116718103]
  Reinforcement learning post-training unlocks complex reasoning in LLMs.<n> benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks.<n>We challenge this activation-centered account by separating activation from control.
  arXiv  Detail & Related papers  (2026-08-08T16:44:47Z)
• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Distributionally Robust Imitation Learning: Layered Control Architecture for Certifiable Autonomy [24.387204607890578]
  Imitation learning (IL) enables autonomous behavior by learning from expert demonstrations.<n>We propose a Layered Control Architecture (LCA) that integrates TaSIL andellonedrac.<n>Our solution paves the path for designing fully certifiable autonomy pipelines.
  arXiv  Detail & Related papers  (2025-12-19T18:58:11Z)
• Comparing Behavioural Cloning and Reinforcement Learning for Spacecraft Guidance and Control Networks [4.868863044142366]
  Guidance & control networks (G&CNETs) provide a promising alternative to on-board guidance and control (G&C) architectures for spacecraft.<n>When training G&CNETs, two predominant paradigms emerge: behavioural cloning (BC), which mimics optimal trajectories, and reinforcement learning (RL), which learns optimal behaviour through trials and errors.<n>We conduct a systematic evaluation of BC and RL specifically for training G&CNETs on continuous-thrust spacecraft trajectory optimisation tasks.<n>We introduce a novel RL training framework tailored to G&CNETs, incorporating decoupled action and control frequencies alongside reward
  arXiv  Detail & Related papers  (2025-07-22T07:43:38Z)
• Diffusion Guidance Is a Controllable Policy Improvement Operator [98.11511661904618]
  CFGRL is trained with the simplicity of supervised learning, yet can further improve on the policies in the data.<n>On offline RL tasks, we observe a reliable trend -- increased guidance weighting leads to increased performance.
  arXiv  Detail & Related papers  (2025-05-29T14:06:50Z)
• Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective [22.267502760766398]
  Generative Adversarial Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with policy (ST-GAIL)
  arXiv  Detail & Related papers  (2023-12-18T14:04:51Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• Controlled Descent Training [0.0]
  A novel and model-based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. The applicability of the method is demonstrated on standard regression and classification problems.
  arXiv  Detail & Related papers  (2023-03-16T10:45:24Z)
• Actor-Critic based Improper Reinforcement Learning [61.430513757337486]
  We consider an improper reinforcement learning setting where a learner is given $M$ base controllers for an unknown Markov decision process. We propose two algorithms: (1) a Policy Gradient-based approach; and (2) an algorithm that can switch between a simple Actor-Critic scheme and a Natural Actor-Critic scheme.
  arXiv  Detail & Related papers  (2022-07-19T05:55:02Z)
• Parallelized Reverse Curriculum Generation [62.25453821794469]
  For reinforcement learning, it is challenging for an agent to master a task that requires a specific series of actions due to sparse rewards. reverse curriculum generation (RCG) provides a reverse expansion approach that automatically generates a curriculum for the agent to learn. We propose a parallelized approach that simultaneously trains multiple AC pairs and periodically exchanges their critics.
  arXiv  Detail & Related papers  (2021-08-04T15:58:35Z)
• Collision-Free Flocking with a Dynamic Squad of Fixed-Wing UAVs Using Deep Reinforcement Learning [2.555094847583209]
  We deal with the decentralized leader-follower flocking control problem through deep reinforcement learning (DRL) We propose a novel reinforcement learning algorithm CACER-II for training a shared control policy for all the followers. As a result, the variable-length system state can be encoded into a fixed-length embedding vector, which makes the learned DRL policies independent with the number or the order of followers.
  arXiv  Detail & Related papers  (2021-01-20T11:23:35Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)
• Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descent [11.329376606876101]
  We propose to stabilize GAN training via a novel particle-based variational inference -- Langevin Stein variational descent gradient (LSVGD) We show that LSVGD dynamics has an implicit regularization which is able to enhance particles' spread-out and diversity.
  arXiv  Detail & Related papers  (2020-04-22T11:20:04Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)
• On Computation and Generalization of Generative Adversarial Imitation Learning [134.17122587138897]
  Generative Adversarial Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies. This paper investigates the theoretical properties of GAIL.
  arXiv  Detail & Related papers  (2020-01-09T00:40:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.