Planning with affordances: Integrating learned affordance models and symbolic planning

• URL: http://arxiv.org/abs/2502.02768v1
• Date: Tue, 04 Feb 2025 23:15:38 GMT
• Title: Planning with affordances: Integrating learned affordance models and symbolic planning
• Authors: Rajesh Mangannavar,
• Abstract summary: We augment an existing task and motion planning framework with learned affordance models of objects in the world. Each task can be seen as changing the current state of the world to a given goal state. A symbolic planning algorithm uses this information and the starting and goal state to create a feasible plan to reach the desired goal state.
• Score: 0.0
• License:
• Abstract: Intelligent agents working in real-world environments must be able to learn about the environment and its capabilities which enable them to take actions to change to the state of the world to complete a complex multi-step task in a photorealistic environment. Learning about the environment is especially important to perform various multiple-step tasks without having to redefine an agent's action set for different tasks or environment settings. In our work, we augment an existing task and motion planning framework with learned affordance models of objects in the world to enable planning and executing multi-step tasks using learned models. Each task can be seen as changing the current state of the world to a given goal state. The affordance models provide us with what actions are possible and how to perform those actions in any given state. A symbolic planning algorithm uses this information and the starting and goal state to create a feasible plan to reach the desired goal state to complete a given task. We demonstrate our approach in a virtual 3D photorealistic environment, AI2-Thor, and evaluate it on real-world tasks. Our results show that our agent quickly learns how to interact with the environment and is well prepared to perform tasks such as "Moving an object out of the way to reach the desired location."

Related papers

• Adaptation of Task Goal States from Prior Knowledge [1.098383730564372]
  We define a framework to define a task with freedom and variability in its goal state. A robot could use this to observe the execution of a task and target a different goal from the observed one.
  arXiv  Detail & Related papers  (2025-02-06T09:51:04Z)
• CLIMB: Language-Guided Continual Learning for Task Planning with Iterative Model Building [30.274897468701592]
  We present CLIMB, a continual learning framework for robot task planning. CLIMB builds a model from a natural language description, learn non-obvious predicates while solving tasks, and store that information for future problems. We also develop the BlocksWorld++ domain, a simulated environment with an easily usable real counterpart, together with a curriculum of tasks with progressing difficulty for evaluating continual learning.
  arXiv  Detail & Related papers  (2024-10-17T16:53:43Z)
• Flex: End-to-End Text-Instructed Visual Navigation with Foundation Models [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies. Our findings are synthesized in Flex (Fly-lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors. We demonstrate the effectiveness of this approach on quadrotor fly-to-target tasks, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Embodied Instruction Following in Unknown Environments [66.60163202450954]
  We propose an embodied instruction following (EIF) method for complex tasks in the unknown environment. We build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller. For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues.
  arXiv  Detail & Related papers  (2024-06-17T17:55:40Z)
• Embodied Task Planning with Large Language Models [86.63533340293361]
  We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin.
  arXiv  Detail & Related papers  (2023-07-04T17:58:25Z)
• MANER: Multi-Agent Neural Rearrangement Planning of Objects in Cluttered Environments [8.15681999722805]
  This paper proposes a learning-based framework for multi-agent object rearrangement planning. It addresses the challenges of task sequencing and path planning in complex environments.
  arXiv  Detail & Related papers  (2023-06-10T23:53:28Z)
• Egocentric Planning for Scalable Embodied Task Achievement [6.870094263016224]
  Egocentric Planning is an innovative approach that combines symbolic planning and Object-oriented POMDPs to solve tasks in complex environments. We evaluated our approach in ALFRED, a simulated environment designed for domestic tasks, and demonstrated its high scalability. Our method requires reliable perception and the specification or learning of a symbolic description of the preconditions and effects of the agent's actions.
  arXiv  Detail & Related papers  (2023-06-02T06:41:24Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• PackIt: A Virtual Environment for Geometric Planning [68.79816936618454]
  PackIt is a virtual environment to evaluate and potentially learn the ability to do geometric planning. We construct a set of challenging packing tasks using an evolutionary algorithm.
  arXiv  Detail & Related papers  (2020-07-21T22:51:17Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.