Active Reward Machine Inference From Raw State Trajectories

• URL: http://arxiv.org/abs/2604.07480v1
• Date: Wed, 08 Apr 2026 18:19:55 GMT
• Title: Active Reward Machine Inference From Raw State Trajectories
• Abstract summary: This paper deals with the problem of learning reward machines directly from raw state and policy information.<n>As opposed to existing works, we assume no access to observations of rewards, labels, or machine nodes, and show what trajectory data is sufficient for learning the reward machine in this information-scarce regime.<n>We then extend the result to an active learning setting where we incrementally query trajectory extensions to improve data (and indirectly computational) efficiency.
• Score: 0.1274452325287335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward machines are automaton-like structures that capture the memory required to accomplish a multi-stage task. When combined with reinforcement learning or optimal control methods, they can be used to synthesize robot policies to achieve such tasks. However, specifying a reward machine by hand, including a labeling function capturing high-level features that the decisions are based on, can be a daunting task. This paper deals with the problem of learning reward machines directly from raw state and policy information. As opposed to existing works, we assume no access to observations of rewards, labels, or machine nodes, and show what trajectory data is sufficient for learning the reward machine in this information-scarce regime. We then extend the result to an active learning setting where we incrementally query trajectory extensions to improve data (and indirectly computational) efficiency. Results are demonstrated with several grid world examples.

Related papers

• Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Robot Learning [62.3886343725955]
  We introduce a novel value-based reinforcement learning algorithm that learns a critic network that outputs Q-values over a sequence of actions.<n>Experiments show that CQN-AS outperforms several baselines on a variety of sparse-reward humanoid control and tabletop manipulation tasks.
  arXiv  Detail & Related papers  (2024-11-19T01:23:52Z)
• A Generalized Acquisition Function for Preference-based Reward Learning [12.158619866176487]
  Preference-based reward learning is a popular technique for teaching robots and autonomous systems how a human user wants them to perform a task. Previous works have shown that actively synthesizing preference queries to maximize information gain about the reward function parameters improves data efficiency. We show that it is possible to optimize for learning the reward function up to a behavioral equivalence class, such as inducing the same ranking over behaviors, distribution over choices, or other related definitions of what makes two rewards similar.
  arXiv  Detail & Related papers  (2024-03-09T20:32:17Z)
• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)
• A Reinforcement Learning Approach for Robotic Unloading from Visual Observations [1.420663986837751]
  In this work, we focus on a robotic unloading problem from visual observations. We propose a hierarchical controller structure that combines a high-level decision-making module with classical motion control. Our experiments demonstrate that both these elements play a crucial role in achieving improved learning performance.
  arXiv  Detail & Related papers  (2023-09-12T22:22:28Z)
• RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation [36.43143326197769]
  Track-Any-Point (TAP) models isolate the relevant motion in a demonstration, and parameterize a low-level controller to reproduce this motion across changes in the scene configuration. We show this results in robust robot policies that can solve complex object-arrangement tasks such as shape-matching, stacking, and even full path-following tasks such as applying glue and sticking objects together.
  arXiv  Detail & Related papers  (2023-08-30T11:57:04Z)
• ALBench: A Framework for Evaluating Active Learning in Object Detection [102.81795062493536]
  This paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols.
  arXiv  Detail & Related papers  (2022-07-27T07:46:23Z)
• Diverse Complexity Measures for Dataset Curation in Self-driving [80.55417232642124]
  We propose a new data selection method that exploits a diverse set of criteria that quantize interestingness of traffic scenes. Our experiments show that the proposed curation pipeline is able to select datasets that lead to better generalization and higher performance.
  arXiv  Detail & Related papers  (2021-01-16T23:45:02Z)
• Disentangled Planning and Control in Vision Based Robotics via Reward Machines [13.486750561133634]
  We augment a Deep Q-Learning agent with a Reward Machine (DQRM) to increase speed of learning vision-based policies for robot tasks. A reward machine (RM) is a finite state machine that decomposes a task into a discrete planning graph and equips the agent with a reward function to guide it toward task completion.
  arXiv  Detail & Related papers  (2020-12-28T19:54:40Z)
• A Framework for Efficient Robotic Manipulation [79.10407063260473]
  We show that a single robotic arm can learn sparse-reward manipulation policies from pixels. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels.
  arXiv  Detail & Related papers  (2020-12-14T22:18:39Z)
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning [78.13740204156858]
  We show that we can reuse prior data to extend new skills simply through dynamic programming. We demonstrate the effectiveness of our approach by chaining together several behaviors seen in prior datasets for solving a new task. We train our policies in an end-to-end fashion, mapping high-dimensional image observations to low-level robot control commands.
  arXiv  Detail & Related papers  (2020-10-27T17:57:29Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)
• Low Dimensional State Representation Learning with Reward-shaped Priors [7.211095654886105]
  We propose a method that aims at learning a mapping from the observations into a lower-dimensional state space. This mapping is learned with unsupervised learning using loss functions shaped to incorporate prior knowledge of the environment and the task. We test the method on several mobile robot navigation tasks in a simulation environment and also on a real robot.
  arXiv  Detail & Related papers  (2020-07-29T13:00:39Z)
• How Useful is Self-Supervised Pretraining for Visual Tasks? [133.1984299177874]
  We evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows.
  arXiv  Detail & Related papers  (2020-03-31T16:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.