Discrete Action Space as a Prerequisite for GRPO Convergence in Small-Model Continuous Control

• URL: http://arxiv.org/abs/2607.21626v1
• Date: Tue, 07 Jul 2026 17:14:19 GMT
• Title: Discrete Action Space as a Prerequisite for GRPO Convergence in Small-Model Continuous Control
• Abstract summary: Group Relative Policy Optimization can fine-tune small language models for simulated quadrotor continuous-control tasks.<n>In our benchmark, vanilla GRPO fine-tuning of Qwen-0.5B for 25 Hz quadrotor velocity control collapses to the trivial zero action.<n>A high-fidelity simulation using Crazyflie 2.1 dynamics surfaces a hover-region training-distribution gap.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We study whether Group Relative Policy Optimization (GRPO) can fine-tune small language models for simulated quadrotor continuous-control tasks. In our benchmark, vanilla GRPO fine-tuning of Qwen-0.5B for 25 Hz quadrotor velocity control collapses to the trivial zero action: 0 percent success rate, with entropy falling from 0.35 to 0.03 within 60 steps. Two ablations - removing the jerk-penalty term and removing the KL anchor to the pretrained prior - each prevent entropy collapse, yet neither enables learning. When the action interface is replaced by a 5-way categorical choice over PID presets, training converges. The resulting controller traces a smoothness-reliability Pareto frontier along training duration; both endpoints are reported: 98.6 percent success with 0.656 m/s3 jerk at 64 steps, and 100 percent success with 1.103 m/s3 jerk, or 0.796 under a matched velocity cap, at 256 steps. The recipe is evaluated across three pretrained language models. As context, a re-tuned classical baseline, PID with Ki = 0.30 and vmax = 2.5, reaches the same 100 percent success rate at jerk 0.736 m/s3. A high-fidelity simulation using Crazyflie 2.1 dynamics surfaces a hover-region training-distribution gap.

Related papers

• WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training [48.290951447286744]
  WDL-OPD is a mixture-constrained co-training method with two trainable policies.<n>An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL.<n>We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
  arXiv  Detail & Related papers  (2026-08-10T11:22:53Z)
• SoRoMoX: Fast, Differentiable, and Parallelizable Soft Robot Models [40.64972837908431]
  SoRoMoX implements articulated, Piecewise Constant Strain, and Variable Strain models through a unified, control-ready interface.<n> Sequential CPU rollouts are up to 18.1x faster than state-of-the-art alternatives, while GPU-parallel rollouts increase throughput by up to 234.6x.
  arXiv  Detail & Related papers  (2026-08-06T23:31:47Z)
• Hermite Curves as Trajectory Priors for Vision-Language-Action Models [41.51953284790607]
  We introduce Hermite trajectory priors, parameterizing the chunk trajectory as a piecewise cubic Hermite curve defined by endpoint positions and velocities.<n>We instantiate this fixed operator across discrete autoregressive and continuous generative paradigms.<n>Across simulation benchmarks and real-robot platforms, Hermite Regularization achieves superior performance.
  arXiv  Detail & Related papers  (2026-08-02T14:18:17Z)
• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation [17.919178151901573]
  We show that standard on-policy distillation (OPD) for large language models suffers from severe training pathologies in practice.<n>PowerOPD is a sign-consistent reward from the Box-Cox power transformation, parameterized by alpha > 0, of which the log-ratio is the degenerate alpha -> 0 limit.<n>It achieves benchmark-averaged Avg@8/Pass@8 gains of up to +6.37/+5.71 over vanilla OPD, +3.01/+3.54 over post-hoc stabilization, and +2.59/+8.90 over full-vocabulary OPD.
  arXiv  Detail & Related papers  (2026-06-15T18:37:51Z)
• TASR: Training-Free Adaptive Stopping for Iterative Retrieval [12.177557521540082]
  Iterative retrieval-augmented generation agents commonly overspend by continuing to retrieve after the model has converged on an answer.<n>We propose TASR (Training-Free Adaptive Stopping Rule), a one-line predicate that fires when the model repeats its previous-round normalized answer.
  arXiv  Detail & Related papers  (2026-06-11T18:35:14Z)
• Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models [0.0]
  Group Relative Policy optimisation has emerged as an effective reinforcement-learning algorithm for aligning language models on reasoning tasks.<n>We introduce two complementary extensions: Adaptive-Horizon GRPO and Selective-Advantage AH-GRPO.<n>Our analysis shows that asymmetric discounting preserves the full gradient signal on correct solutions, prevents entropy collapse, and substantially stabilises training.
  arXiv  Detail & Related papers  (2026-06-03T20:57:57Z)
• ESPO: Early-Stopping Proximal Policy Optimization [78.79610718910628]
  ESPO (Early-Stopping Proximal Policy Optimization) detects trajectory failure on-the-fly and terminates rollouts early.<n>On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME2024 (46.28% vs. 45.25%), AMC2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%)
  arXiv  Detail & Related papers  (2026-05-28T12:40:22Z)
• CrossVLA: Cross-Paradigm Post-Training and Inference Optimization for Vision-Language-Action Models [7.489020109808801]
  We present CrossVLA, an empirical study of cross-paradigm Vision-Language-Action (VLA) post-training.<n>Three contributions: (i) a surrogate flow-matching log-probability estimator that lets DPO operate on continuous-action backbones without probability-flow ODE integration; (ii) a head-to-head comparison of LoRA and DoRA as the parameter-efficient layer for VLA DPO; and (iii) an inference-time anatomy showing the denoise loop dominates 78.6% of sample_actions latency and prefix-K/V caching a la
  arXiv  Detail & Related papers  (2026-05-21T01:02:41Z)
• Predicting Upcoming Stuttering Events from Three-Second Audio: Stratified Evaluation Reveals Severity-Selective Precursors, and the Model Deploys Fully On-Device [0.0]
  We train a CNN on SEP-28k (Apple, 20,131 three-second clips) to predict whether the next contiguous clip contains any disfluency.<n>The aggregate objective converges to a severity-selective predictor because severe events carry prosodic precursors.<n>Cross-population transfer: without fine-tuning, the same checkpoint applied to 1,024 pediatric Children-Who-Stutter utterances attains AUC 0.674 detection and 0.655 prediction.
  arXiv  Detail & Related papers  (2026-04-30T00:30:28Z)
• SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Models [48.335262141752715]
  Post-training pipeline for diffusion models has two stages: supervised fine-tuning (SFT) on curated data and reinforcement learning (RL) with reward models.<n>We propose SOAR (Self-Correction for Optimal Alignment and Refinement), a bias-correction post-training method that fills this gap.<n>It is on-policy, reward-free, and provides dense per-time supervision with no credit-assignment problem.
  arXiv  Detail & Related papers  (2026-04-14T11:45:15Z)
• Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation [95.89924101984566]
  We introduce OptimusVLA, a dual-memory VLA framework with Global Prior Memory (GPM) and Local Consistency Memory (LCM)<n>GPM replaces Gaussian noise with task-level priors retrieved from semantically similar trajectories.<n>LCM injects a learned consistency constraint that enforces temporal coherence and smoothness of trajectory.
  arXiv  Detail & Related papers  (2026-02-22T15:39:34Z)
• BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning [73.46118996284888]
  Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence.<n>We propose BadCLIP++, a unified framework that tackles both challenges.<n>For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions.<n>For persistence, we stabilize trigger embeddings via radius shrinkage and centroid alignment.
  arXiv  Detail & Related papers  (2026-02-19T08:31:16Z)
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model [56.92219181993453]
  We propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix) to enable on-policyRFT methods like PPO and GRPO to leverage off-policy data.<n>ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio for efficient training; (2) KL-Convex policy constraint to balance the trade-off between stability and flexibility; (3) Policy reincarnation to achieve a seamless transition from efficient early-stage learning to steady improvement.
  arXiv  Detail & Related papers  (2025-07-09T14:29:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.