Nonuniqueness and Convergence to Equivalent Solutions in Observer-based Inverse Reinforcement Learning

• URL: http://arxiv.org/abs/2210.16299v4
• Date: Thu, 30 May 2024 17:31:41 GMT
• Title: Nonuniqueness and Convergence to Equivalent Solutions in Observer-based Inverse Reinforcement Learning
• Authors: Jared Town, Zachary Morrison, Rushikesh Kamalapurkar,
• Abstract summary: Key challenge in solving the deterministic inverse reinforcement learning (IRL) problem online and in real-time is the existence of multiple solutions. Nonuniqueness necessitates the study of the notion of equivalent solutions. Regularized history stack observer that converges to approximately equivalent solutions of the IRL problem is developed.
• Score: 0.8943924354248618
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A key challenge in solving the deterministic inverse reinforcement learning (IRL) problem online and in real-time is the existence of multiple solutions. Nonuniqueness necessitates the study of the notion of equivalent solutions, i.e., solutions that result in a different cost functional but same feedback matrix, and convergence to such solutions. While offline algorithms that result in convergence to equivalent solutions have been developed in the literature, online, real-time techniques that address nonuniqueness are not available. In this paper, a regularized history stack observer that converges to approximately equivalent solutions of the IRL problem is developed. Novel data-richness conditions are developed to facilitate the analysis and simulation results are provided to demonstrate the effectiveness of the developed technique.

Related papers

• Learning the Boundary of Solvability: Aligning LLMs to Detect Unsolvable Problems [51.62477754641947]
  We propose UnsolvableQA and UnsolvableRL to solve feasible problems, detect inherent contradictions, and prudently refuse tasks beyond capability.<n>Specifically, we construct UnsolvableQA, a dataset of paired solvable and unsolvable instances derived via a dual-track methodology.<n>Building on this dataset, we introduce UnsolvableRL, a reinforcement learning framework with three reward components jointly accounting for accuracy, unsolvability, and difficulty.
  arXiv  Detail & Related papers  (2025-12-01T13:32:59Z)
• Online Inference of Constrained Optimization: Primal-Dual Optimality and Sequential Quadratic Programming [55.848340925419286]
  We study online statistical inference for the solutions of quadratic optimization problems with equality and inequality constraints.<n>We develop a sequential programming (SSQP) method to solve these problems, where the step direction is computed by sequentially performing an approximation of the objective and a linear approximation of the constraints.<n>We show that our method global almost moving-average convergence and exhibits local normality with an optimal primal-dual limiting matrix in the sense of Hjek and Le Cam.
  arXiv  Detail & Related papers  (2025-11-27T06:16:17Z)
• Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble [10.047968926134363]
  We explore the capability of physics-informed neural networks (PINNs) to discover multiple solutions. PINNs can effectively uncover multiple solutions to nonlinear ordinary and partial differential equations. We propose utilizing PINN-generated solutions as initial conditions or initial guesses for conventional numerical solvers.
  arXiv  Detail & Related papers  (2025-03-08T19:32:22Z)
• Continuous Approximation of the Fully Connected Ising Hamiltonian: Exact Ground State Solutions for a Novel Class of Ising Models with Applications to Fidelity Assessment in Ising Machines [1.2903829793534267]
  We present a novel analytical approach to solving large-scale Ising problems by reformulating the discrete Ising Hamiltonian into a continuous framework. To validate our method, we conducted numerical experiments comparing our analytical solutions with those obtained from a quantum-inspired Ising algorithm and a quantum Ising machine.
  arXiv  Detail & Related papers  (2024-11-29T10:43:23Z)
• Homotopy Continuation Made Easy: Regression-based Online Simulation of Starting Problem-Solution Pairs [17.543457476766367]
  homotopy continuation has been introduced as a plausible alternative to elimination templates. Our innovation consists of employing a regression network trained in simulation to directly predict a solution from input correspondences. We apply this elegant combination to generalized camera resectioning, and also introduce a new solution to the challenging generalized relative pose and scale problem.
  arXiv  Detail & Related papers  (2024-11-06T08:22:00Z)
• Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spaces [47.907236421762626]
  This work studies discrete-time discounted Markov decision processes with continuous state and action spaces. We first consider the case in which we have access to the entire expert policy and characterize the set of solutions to the inverse problem.
  arXiv  Detail & Related papers  (2024-05-24T12:53:07Z)
• Continuous Tensor Relaxation for Finding Diverse Solutions in Combinatorial Optimization Problems [0.6906005491572401]
  This study introduces Continual Anne Relaxationing (CTRA) for unsupervised-learning (UL)-based CO solvers. CTRA is a computationally efficient framework for finding diverse solutions in a single training run. Numerical experiments show that CTRA enables UL-based solvers to find these diverse solutions much faster than repeatedly running existing solvers.
  arXiv  Detail & Related papers  (2024-02-03T15:31:05Z)
• HomPINNs: homotopy physics-informed neural networks for solving the inverse problems of nonlinear differential equations with multiple solutions [6.89453634946458]
  We propose homotopy physics-informed neural networks (HomPINNs) to solve inverse problems of nonlinear differential equations (DEs) The proposed framework begins with the use of NNs to simultaneously approximate unlabeled observations across diverse solutions while adhering to DE constraints. Our findings demonstrate that the proposed method is scalable and adaptable, providing an effective solution for solving DEs with multiple solutions and unknown parameters.
  arXiv  Detail & Related papers  (2023-04-06T01:20:23Z)
• Learning to Optimize with Stochastic Dominance Constraints [103.26714928625582]
  In this paper, we develop a simple yet efficient approach for the problem of comparing uncertain quantities. We recast inner optimization in the Lagrangian as a learning problem for surrogate approximation, which bypasses apparent intractability. The proposed light-SD demonstrates superior performance on several representative problems ranging from finance to supply chain management.
  arXiv  Detail & Related papers  (2022-11-14T21:54:31Z)
• Learning Proximal Operators to Discover Multiple Optima [66.98045013486794]
  We present an end-to-end method to learn the proximal operator across non-family problems. We show that for weakly-ized objectives and under mild conditions, the method converges globally.
  arXiv  Detail & Related papers  (2022-01-28T05:53:28Z)
• Instance-Dependent Confidence and Early Stopping for Reinforcement Learning [99.57168572237421]
  Various algorithms for reinforcement learning (RL) exhibit dramatic variation in their convergence rates as a function of problem structure. This research provides guarantees that explain textitex post the performance differences observed. A natural next step is to convert these theoretical guarantees into guidelines that are useful in practice.
  arXiv  Detail & Related papers  (2022-01-21T04:25:35Z)
• Discovering Diverse Solutions in Deep Reinforcement Learning [84.45686627019408]
  Reinforcement learning algorithms are typically limited to learning a single solution of a specified task. We propose an RL method that can learn infinitely many solutions by training a policy conditioned on a continuous or discrete low-dimensional latent variable.
  arXiv  Detail & Related papers  (2021-03-12T04:54:31Z)
• Reversible Action Design for Combinatorial Optimization with Reinforcement Learning [35.50454156611722]
  Reinforcement learning (RL) has recently emerged as a new framework to tackle these problems. We propose a general RL framework that not only exhibits state-of-the-art empirical performance but also generalizes to a variety class of COPs.
  arXiv  Detail & Related papers  (2021-02-14T18:05:42Z)
• Dynamic Federated Learning [57.14673504239551]
  Federated learning has emerged as an umbrella term for centralized coordination strategies in multi-agent environments. We consider a federated learning model where at every iteration, a random subset of available agents perform local updates based on their data. Under a non-stationary random walk model on the true minimizer for the aggregate optimization problem, we establish that the performance of the architecture is determined by three factors, namely, the data variability at each agent, the model variability across all agents, and a tracking term that is inversely proportional to the learning rate of the algorithm.
  arXiv  Detail & Related papers  (2020-02-20T15:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.