Quantifying the Effect of Feedback Frequency in Interactive Reinforcement Learning for Robotic Tasks

• URL: http://arxiv.org/abs/2207.09845v1
• Date: Wed, 20 Jul 2022 12:17:02 GMT
• Title: Quantifying the Effect of Feedback Frequency in Interactive Reinforcement Learning for Robotic Tasks
• Abstract summary: Reinforcement learning has become widely adopted in robot control. Interactive feedback has been shown to speed up RL considerably. There is no consensus about which feedback frequency is optimal.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has become widely adopted in robot control. Despite many successes, one major persisting problem can be very low data efficiency. One solution is interactive feedback, which has been shown to speed up RL considerably. As a result, there is an abundance of different strategies, which are, however, primarily tested on discrete grid-world and small scale optimal control scenarios. In the literature, there is no consensus about which feedback frequency is optimal or at which time the feedback is most beneficial. To resolve these discrepancies we isolate and quantify the effect of feedback frequency in robotic tasks with continuous state and action spaces. The experiments encompass inverse kinematics learning for robotic manipulator arms of different complexity. We show that seemingly contradictory reported phenomena occur at different complexity levels. Furthermore, our results suggest that no single ideal feedback frequency exists. Rather that feedback frequency should be changed as the agent's proficiency in the task increases.

Related papers

• Learning High-Frequency Continuous Action Chunks in Latent Space [10.321199672785392]
  Modern robotic policies increasingly rely on action chunking to execute complex tasks in the physical world.<n>At such high frequencies, policies often fail to generate actions that are both temporally smooth and spatially consistent.<n>We address this challenge by shifting high-frequency action learning from the action space to a latent space with variational autoencoder (VAE)<n>This formulation significantly improves both temporal and spatial consistency of high-frequency control.
  arXiv  Detail & Related papers  (2026-05-24T08:22:14Z)
• Probing Embodied LLMs: When Higher Observation Fidelity Hurts Problem Solving [7.648784748888186]
  Large Language Models are proposed as cognitive components for robotic systems.<n>We study embodied LLM agents behaviorally by varying the information available to the agent.<n>We find that moderate noise improves performance, peaking at a 40% flip probability with a 2.85-fold success rate increase over the noise-free baseline.
  arXiv  Detail & Related papers  (2026-05-19T16:27:00Z)
• When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoning [39.871701555686634]
  We study how the interaction between researchers and agents affect the results using SCALAR (Structured Critic--Actor Loop for AI Reasoning)<n>The Actor proposes solutions, the Critic provides iterative feedback, and an independent Judge evaluates the transcript against reference solutions.<n>We vary the Actor persona, the Critic feedback strategy, and the Actor model family and scale.
  arXiv  Detail & Related papers  (2026-05-07T18:00:01Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning [58.80217284841095]
  Multi-turn problem solving is critical yet challenging for Large Reasoning Models (LRMs) to reflect on their reasoning and revise from feedback.<n>Existing Reinforcement Learning (RL) methods train large reasoning models on a single-turn paradigm with verifiable rewards.<n>We introduce Unary Feedback as Observation (UFO) for reinforcement learning, which uses minimal yet common unary user feedback during iterative problem solving.
  arXiv  Detail & Related papers  (2025-07-18T18:07:38Z)
• Reinforcement Learning with Segment Feedback [56.54271464134885]
  We consider a model named RL with segment feedback, which offers a general paradigm filling the gap between per-state-action feedback and trajectory feedback. Under binary feedback, increasing the number of segments $m$ decreases the regret at an exponential rate; in contrast, surprisingly, under sum feedback, increasing $m$ does not reduce the regret significantly.
  arXiv  Detail & Related papers  (2025-02-03T23:08:42Z)
• Reinforcement Learning with Action Sequence for Data-Efficient Robot Learning [62.3886343725955]
  We introduce a novel RL algorithm that learns a critic network that outputs Q-values over a sequence of actions. By explicitly training the value functions to learn the consequence of executing a series of current and future actions, our algorithm allows for learning useful value functions from noisy trajectories.
  arXiv  Detail & Related papers  (2024-11-19T01:23:52Z)
• REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation [61.7171775202833]
  We introduce an efficient system for learning dexterous manipulation skills withReinforcement learning. The main idea of our approach is the integration of recent advances in sample-efficient RL and replay buffer bootstrapping. Our system completes the real-world training cycle by incorporating learned resets via an imitation-based pickup policy.
  arXiv  Detail & Related papers  (2023-09-06T19:05:31Z)
• Primitive Skill-based Robot Learning from Human Evaluative Feedback [28.046559859978597]
  Reinforcement learning algorithms face challenges when dealing with long-horizon robot manipulation tasks in real-world environments. We propose a novel framework, SEED, which leverages two approaches: reinforcement learning from human feedback (RLHF) and primitive skill-based reinforcement learning. Our results show that SEED significantly outperforms state-of-the-art RL algorithms in sample efficiency and safety.
  arXiv  Detail & Related papers  (2023-07-28T20:48:30Z)
• Dynamic Decision Frequency with Continuous Options [11.83290684845269]
  In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. We propose a framework called Continuous-Time Continuous-Options (CTCO) where the agent chooses options as sub-policies of variable durations. We show that our algorithm's performance is not affected by the choice of environment interaction frequency.
  arXiv  Detail & Related papers  (2022-12-06T19:51:12Z)
• Offline Reinforcement Learning at Multiple Frequencies [62.08749079914275]
  We study how well offline reinforcement learning algorithms can accommodate data with a mixture of frequencies during training. We present a simple yet effective solution that enforces consistency in the rate of $Q$-value updates to stabilize learning.
  arXiv  Detail & Related papers  (2022-07-26T17:54:49Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention [67.1936055742498]
  We show that multi-task learning can effectively scale reset-free learning schemes to much more complex problems. This work shows the ability to learn dexterous manipulation behaviors in the real world with RL without any human intervention.
  arXiv  Detail & Related papers  (2021-04-22T17:38:27Z)
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation [115.4071729927011]
  We study the effects of using mid-level visual representations as generic and easy-to-decode perceptual state in an end-to-end RL framework. We show that they aid generalization, improve sample complexity, and lead to a higher final performance. In practice, this means that mid-level representations could be used to successfully train policies for tasks where domain randomization and learning-from-scratch failed.
  arXiv  Detail & Related papers  (2020-11-13T00:16:05Z)
• Robust Learning with Frequency Domain Regularization [1.370633147306388]
  We introduce a new regularization method by constraining the frequency spectra of the filter of the model. We demonstrate the effectiveness of our regularization by (1) defensing to adversarial perturbations; (2) reducing the generalization gap in different architecture; and (3) improving the generalization ability in transfer learning scenario without fine-tune.
  arXiv  Detail & Related papers  (2020-07-07T07:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.