SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy

• URL: http://arxiv.org/abs/2609.12749v1
• Date: Fri, 11 Sep 2026 11:59:52 GMT
• Title: SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
• Abstract summary: We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces the standard log-entropy term with a sigmoid-bounded formulation that stays strictly positive.<n>SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual benchmarks.<n>A direct clipping intervention removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement.
• Score: 2.299479454443024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline-to-online reinforcement learning reduces interaction cost for real-world robot learning but suffers from persistent value estimation instability. Existing methods address this through pessimistic regularization, lower-bound calibration, and architectural normalization, but an overlooked source of instability lies in the entropy formulation: the standard log-entropy term can become negative, destabilizing policy updates. We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces this term with a sigmoid-bounded formulation that stays strictly positive. SCQ retains conservative Q regularization and return-based lower-bound calibration, stabilizing policy optimization without sacrificing exploration. We evaluate SCQ on D4RL (Minari) benchmarks under both single-demonstration and standard dataset settings, as well as on simulation and real-world visual tasks. SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual benchmarks, and transfers to four real-robot platforms including manipulation, wheeled, quadruped, and humanoid systems. A direct clipping intervention that removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement. Project website: https://scq-rl.github.io.

Related papers

• Revisiting TD Target Aggregation under Uncertainty in Q-Learning [2.4908682918195804]
  Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy aggregation over next-state values.<n>We propose textbfSuccessor Rollout textbfAggregation textbfDeep textbfQ-Network (SADQ), a simple rollout to Q-learning that regularizes how the TD target is formed.
  arXiv  Detail & Related papers  (2026-08-04T03:31:24Z)
• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• From Robotics to Sepsis Treatment: Offline RL via Geometric Pessimism [0.0]
  Methods like CQL offer rigorous conservatism but require tremendous compute power.<n>IQL often fail to correct OOD errors on pathological datasets, collapsing to Behavioural Cloning.<n>We propose Geometric Pessimism, a compute-efficient framework that augments standard IQL with density-based penalty.
  arXiv  Detail & Related papers  (2026-02-09T13:48:25Z)
• Trainability-Oriented Hybrid Quantum Regression via Geometric Preconditioning and Curriculum Optimization [0.6445605125467574]
  Quantum neural networks (QNNs) have attracted growing interest for scientific machine learning.<n>In regression settings they often suffer from limited trainability under noisy gradients and ill-conditioned optimization.<n>We propose a hybrid quantum-classical regression framework designed to mitigate these bottlenecks.
  arXiv  Detail & Related papers  (2026-01-17T07:32:18Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction [3.2883573376133555]
  Deep Q-Networks (DQNs) estimate future returns by learning from transitions sampled from a replay buffer.<n>SADQ integrates successor-state distributions into the Q-value estimation process.<n>We provide theoretical guarantees that SADQ maintains unbiased value estimates while reducing training variance.
  arXiv  Detail & Related papers  (2025-11-05T20:04:53Z)
• Relative Entropy Pathwise Policy Optimization [66.03329137921949]
  We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories.<n>We show how to combine policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning.
  arXiv  Detail & Related papers  (2025-07-15T06:24:07Z)
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning [46.67828766038463]
  We propose a framework, Adaptive Conservative Level in Q-Learning (ACL-QL), which limits the Q-values in a mild range.<n>ACL-QL enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions.<n>Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition.
  arXiv  Detail & Related papers  (2024-12-22T04:18:02Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Strategically Conservative Q-Learning [89.17906766703763]
  offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions. We propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate.
  arXiv  Detail & Related papers  (2024-06-06T22:09:46Z)
• Mildly Conservative Q-Learning for Offline Reinforcement Learning [63.2183622958666]
  offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. Existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values.
  arXiv  Detail & Related papers  (2022-06-09T19:44:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.