Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors

• URL: http://arxiv.org/abs/2006.13205v2
• Date: Fri, 27 Nov 2020 22:34:30 GMT
• Title: Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Authors: Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, Sergey Levine
• Abstract summary: The ability to predict and plan into the future is fundamental for agents acting in the world. Current learning approaches for visual prediction and planning fail on long-horizon tasks. We propose a framework for visual prediction and planning that is able to overcome both of these limitations.
• Score: 124.30562402952319
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The ability to predict and plan into the future is fundamental for agents acting in the world. To reach a faraway goal, we predict trajectories at multiple timescales, first devising a coarse plan towards the goal and then gradually filling in details. In contrast, current learning approaches for visual prediction and planning fail on long-horizon tasks as they generate predictions (1) without considering goal information, and (2) at the finest temporal resolution, one step at a time. In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations. First, we formulate the problem of predicting towards a goal and propose the corresponding class of latent space goal-conditioned predictors (GCPs). GCPs significantly improve planning efficiency by constraining the search space to only those trajectories that reach the goal. Further, we show how GCPs can be naturally formulated as hierarchical models that, given two observations, predict an observation between them, and by recursively subdividing each part of the trajectory generate complete sequences. This divide-and-conquer strategy is effective at long-term prediction, and enables us to design an effective hierarchical planning algorithm that optimizes trajectories in a coarse-to-fine manner. We show that by using both goal-conditioning and hierarchical prediction, GCPs enable us to solve visual planning tasks with much longer horizon than previously possible.

Related papers

• Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos [48.15438373870542]
  VidAssist is an integrated framework designed for zero/few-shot goal-oriented planning in instructional videos. It employs a breadth-first search algorithm for optimal plan generation. Experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning setups.
  arXiv  Detail & Related papers  (2024-09-30T17:57:28Z)
• Progressive Pretext Task Learning for Human Trajectory Prediction [44.07301075351432]
  We introduce a novel Progressive Pretext Task learning (PPT) framework, which progressively enhances the model's capacity of capturing short-term dynamics and long-term dependencies. We design a Transformer-based trajectory predictor, which is able to achieve highly efficient two-step reasoning.
  arXiv  Detail & Related papers  (2024-07-16T10:48:18Z)
• G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction [23.181232260820373]
  We propose G2LTraj, a global-to-local generation approach for trajectory prediction. We generate a series of global key steps that uniformly cover the entire future time range. In this way, we prevent the accumulated error from propagating beyond the adjacent key steps.
  arXiv  Detail & Related papers  (2024-04-30T07:53:34Z)
• Imitating Graph-Based Planning with Goal-Conditioned Policies [72.61631088613048]
  We present a self-imitation scheme which distills a subgoal-conditioned policy into the target-goal-conditioned policy. We empirically show that our method can significantly boost the sample-efficiency of the existing goal-conditioned RL methods.
  arXiv  Detail & Related papers  (2023-03-20T14:51:10Z)
• PALMER: Perception-Action Loop with Memory for Long-Horizon Planning [1.5469452301122177]
  We introduce a general-purpose planning algorithm called PALMER. Palmer combines classical sampling-based planning algorithms with learning-based perceptual representations. This creates a tight feedback loop between representation learning, memory, reinforcement learning, and sampling-based planning.
  arXiv  Detail & Related papers  (2022-12-08T22:11:49Z)
• DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets [29.239524389784606]
  We propose an anchor-free and end-to-end trajectory prediction model, named DenseTNT, that directly outputs a set of trajectories from dense goal candidates. DenseTNT achieves state-of-the-art performance, ranking 1st on the Argoverse motion forecasting benchmark and being the 1st place winner of the 2021 Open Motion Prediction Challenge.
  arXiv  Detail & Related papers  (2021-08-22T05:27:35Z)
• Model-Based Reinforcement Learning via Latent-Space Collocation [110.04005442935828]
  We argue that it is easier to solve long-horizon tasks by planning sequences of states rather than just actions. We adapt the idea of collocation, which has shown good results on long-horizon tasks in optimal control literature, to the image-based setting by utilizing learned latent state space models.
  arXiv  Detail & Related papers  (2021-06-24T17:59:18Z)
• From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting [54.273455592965355]
  Uncertainty in future trajectories stems from two sources: (a) sources known to the agent but unknown to the model, such as long term goals and (b)sources that are unknown to both the agent & the model, such as intent of other agents & irreducible randomness indecisions. We model the epistemic un-certainty through multimodality in long term goals and the aleatoric uncertainty through multimodality in waypoints& paths. To exemplify this dichotomy, we also propose a novel long term trajectory forecasting setting, with prediction horizons upto a minute, an order of magnitude longer than prior works.
  arXiv  Detail & Related papers  (2020-12-02T21:01:29Z)
• Goal-driven Long-Term Trajectory Prediction [30.054850639996033]
  We propose to model a hypothetical process that determines pedestrians' goals and the impact of such process on long-term future trajectories. We design Goal-driven Trajectory Prediction model - a dual-channel neural network that realizes such intuition. The model is shown to outperform the state-of-the-art in various settings, especially in large prediction horizons.
  arXiv  Detail & Related papers  (2020-11-05T10:47:33Z)
• The Importance of Prior Knowledge in Precise Multimodal Prediction [71.74884391209955]
  Roads have well defined geometries, topologies, and traffic rules. In this paper we propose to incorporate structured priors as a loss function. We demonstrate the effectiveness of our approach on real-world self-driving datasets.
  arXiv  Detail & Related papers  (2020-06-04T03:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.