Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Role of Bellman Constraints

• URL: http://arxiv.org/abs/2603.22713v1
• Date: Tue, 24 Mar 2026 02:06:31 GMT
• Title: Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Role of Bellman Constraints
• Abstract summary: Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors in behavioral cloning (BC)<n>This paper revisits IQ-Learn and demonstrates that it provably reduces to BC and suffers from an imitation gap lower bound with quadratic dependence on horizon.<n>We introduce a primal-dual framework for distribution matching, yielding a new Q-based IL method, Dual Q-DM.
• Score: 19.446845699075784
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors in behavioral cloning (BC), but often exhibits training instability due to adversarial optimization. To avoid this issue, a class of non-adversarial Q-based imitation learning (IL) methods, represented by IQ-Learn, has emerged and is widely believed to outperform BC by leveraging online environment interactions. However, this paper revisits IQ-Learn and demonstrates that it provably reduces to BC and suffers from an imitation gap lower bound with quadratic dependence on horizon, therefore still suffering from compounding errors. Theoretical analysis reveals that, despite using online interactions, IQ-Learn uniformly suppresses the Q-values for all actions on states uncovered by demonstrations, thereby failing to generalize. To address this limitation, we introduce a primal-dual framework for distribution matching, yielding a new Q-based IL method, Dual Q-DM. The key mechanism in Dual Q-DM is incorporating Bellman constraints to propagate high Q-values from visited states to unvisited ones, thereby achieving generalization beyond demonstrations. We prove that Dual Q-DM is equivalent to AIL and can recover expert actions beyond demonstrations, thereby mitigating compounding errors. To the best of our knowledge, Dual Q-DM is the first non-adversarial IL method that is theoretically guaranteed to eliminate compounding errors. Experimental results further corroborate our theoretical results.

Related papers

• Gated Q-learning: Add Off-Policy Bias to Taste [2.7074235008521246]
  Managing off-policy bias in Q-learning remains a fundamental challenge.<n>Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon.<n> Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons.
  arXiv  Detail & Related papers  (2026-07-31T00:32:46Z)
• Scalable Causal Imitation Learning [48.93769483870838]
  Imitation learning enables learning a policy in an unknown environment with a latent reward signal using expert demonstrations.<n>It struggles when the imitator's and expert's observations are mismatched and unobserved confounders are present in expert demonstrations.<n>We introduce Causal Soft Q Imitation Learning (SQIL) and Causal Inverse soft-Q Learning (IQ-Learn), two off-policy causal imitation learning algorithms.
  arXiv  Detail & Related papers  (2026-07-18T23:33:11Z)
• Agentic Uncertainty Quantification [76.94013626702183]
  We propose a unified Dual-Process Agentic UQ (AUQ) framework that transforms verbalized uncertainty into active, bi-directional control signals.<n>Our architecture comprises two complementary mechanisms: System 1 (Uncertainty-Aware Memory, UAM), which implicitly propagates verbalized confidence and semantic explanations to prevent blind decision-making; and System 2 (Uncertainty-Aware Reflection, UAR), which utilizes these explanations as rational cues to trigger targeted inference-time resolution only when necessary.
  arXiv  Detail & Related papers  (2026-01-22T07:16:26Z)
• Is Q-learning an Ill-posed Problem? [2.4424095531386234]
  This paper investigates the instability of Q-learning in continuous environments.<n>We show that even in relatively simple benchmarks, the fundamental task of Q-learning can be inherently ill-posed and prone to failure.
  arXiv  Detail & Related papers  (2025-02-20T08:42:30Z)
• Suppressing Overestimation in Q-Learning through Adversarial Behaviors [4.36117236405564]
  This paper proposes a new Q-learning algorithm with a dummy adversarial player, which is called dummy adversarial Q-learning (DAQ) The proposed DAQ unifies several Q-learning variations to control overestimation biases, such as maxmin Q-learning and minmax Q-learning. A finite-time convergence of DAQ is analyzed from an integrated perspective by adapting an adversarial Q-learning.
  arXiv  Detail & Related papers  (2023-10-10T03:46:32Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)
• Quantum Imitation Learning [74.15588381240795]
  We propose quantum imitation learning (QIL) with a hope to utilize quantum advantage to speed up IL. We develop two QIL algorithms, quantum behavioural cloning (Q-BC) and quantum generative adversarial imitation learning (Q-GAIL) Experiment results demonstrate that both Q-BC and Q-GAIL can achieve comparable performance compared to classical counterparts.
  arXiv  Detail & Related papers  (2023-04-04T12:47:35Z)
• Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs [50.75812033462294]
  We bridge the gap between practical success of Q-learning and pessimistic theoretical results. We present novel methods Q-Rex and Q-RexDaRe. We show that Q-Rex efficiently finds the optimal policy for linear MDPs.
  arXiv  Detail & Related papers  (2021-10-16T01:47:41Z)
• Self-correcting Q-Learning [14.178899938667161]
  We introduce a new way to address the bias in the form of a "self-correcting algorithm" Applying this strategy to Q-learning results in Self-correcting Q-learning. We show theoretically that this new algorithm enjoys the same convergence guarantees as Q-learning while being more accurate.
  arXiv  Detail & Related papers  (2020-12-02T11:36:24Z)
• ConQUR: Mitigating Delusional Bias in Deep Q-learning [45.21332566843924]
  Delusional bias is a fundamental source of error in approximate Q-learning. We develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "consistent" with the underlying greedy policy class.
  arXiv  Detail & Related papers  (2020-02-27T19:22:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.