StepScorer: Accelerating Reinforcement Learning with Step-wise Scoring and Psychological Regret Modeling

• URL: http://arxiv.org/abs/2602.03171v1
• Date: Tue, 03 Feb 2026 06:39:20 GMT
• Title: StepScorer: Accelerating Reinforcement Learning with Step-wise Scoring and Psychological Regret Modeling
• Abstract summary: This paper introduces the Psychological Regret Model (PRM), a novel approach that accelerates learning by incorporating regret-based feedback signals after each decision step.<n>We demonstrate that PRM achieves stable performance approximately 36% faster than traditional Proximal Policy Optimization (PPO) in benchmark environments such as Lunar Lander.<n>Our results indicate that PRM is particularly effective in continuous control tasks and environments with delayed feedback, making it suitable for real-world applications such as robotics, finance, and adaptive education.
• Score: 4.454674305657783
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Reinforcement learning algorithms often suffer from slow convergence due to sparse reward signals, particularly in complex environments where feedback is delayed or infrequent. This paper introduces the Psychological Regret Model (PRM), a novel approach that accelerates learning by incorporating regret-based feedback signals after each decision step. Rather than waiting for terminal rewards, PRM computes a regret signal based on the difference between the expected value of the optimal action and the value of the action taken in each state. This transforms sparse rewards into dense feedback signals through a step-wise scoring framework, enabling faster convergence. We demonstrate that PRM achieves stable performance approximately 36\% faster than traditional Proximal Policy Optimization (PPO) in benchmark environments such as Lunar Lander. Our results indicate that PRM is particularly effective in continuous control tasks and environments with delayed feedback, making it suitable for real-world applications such as robotics, finance, and adaptive education where rapid policy adaptation is critical. The approach formalizes human-inspired counterfactual thinking as a computable regret signal, bridging behavioral economics and reinforcement learning.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• ADORN: Adaptive Drift handling for Open RAN using Reinforcement Learning [0.9577207941409713]
  Dynamic traffic variations in Open Radio Access Networks (O-RAN) lead to drift.<n>Traditional retraining approaches maintain forecasting accuracy but incur high computational cost.<n>This work proposes a Q-learning-based adaptive retraining approach that formulates the retraining decision as a Markov Decision Process.
  arXiv  Detail & Related papers  (2026-07-09T13:05:42Z)
• FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion [19.265864503552123]
  FastDSAC is a fast and high-performance variant of the Distributional Actor-Critic algorithm for parallel sampling scenarios.<n>We show that FastDSAC stabilizes the overall training process and achieves superior performance and faster convergence compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-30T14:04:26Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• Actor-Critic with Active Importance Sampling [61.41272490044518]
  This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm.<n>AISAC is an extension of the Actor-Critic framework for reducing variance in policy gradient estimation.<n> Experiments show improved learning speed, sample efficiency, and training compared to standard Actor-Critic methods.
  arXiv  Detail & Related papers  (2026-05-08T01:21:32Z)
• Delayed Homomorphic Reinforcement Learning for Environments with Delayed Feedback [11.866061471514582]
  Delayed feedback breaks the Markov assumption and impedes learning and control.<n>We propose a framework grounded in MDP homomorphisms that collapses belief-equivalent augmented states.<n> Experiments on continuous control tasks in MuJoCo benchmark confirm that our algorithm outperforms strong augmentation-based baselines.
  arXiv  Detail & Related papers  (2026-04-04T08:38:52Z)
• Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [29.56905427210088]
  gradient-ARM is a framework that jointly optimize a rubric generator and a judge using reinforcement learning from preference feedback.<n>We show that gradient-ARM achieves state-of-the-art performance among baselines on benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
  arXiv  Detail & Related papers  (2026-02-02T00:50:53Z)
• STARec: An Efficient Agent Framework for Recommender Systems via Autonomous Deliberate Reasoning [54.28691219536054]
  We introduce STARec, a slow-thinking augmented agent framework that endows recommender systems with autonomous deliberative reasoning capabilities.<n>We develop anchored reinforcement training - a two-stage paradigm combining structured knowledge distillation from advanced reasoning models with preference-aligned reward shaping.<n>Experiments on MovieLens 1M and Amazon CDs benchmarks demonstrate that STARec achieves substantial performance gains compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-26T08:47:58Z)
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner [31.033131727230277]
  Large reasoning models (LRMs) have recently shown promise in solving complex math problems when optimized with Reinforcement Learning (RL)<n>We propose a novel intrinsic signal-driven generative process evaluation mechanism operating at the thought level to address major bottlenecks in RL-based training.<n>Experiments on 1.5B and 7B parameter LRMs demonstrate that our method achieves higher problem-solving accuracy with significantly fewer training samples than outcome-only reward baselines.
  arXiv  Detail & Related papers  (2025-07-31T07:54:58Z)
• Noise-based reward-modulated learning [1.0851051226732167]
  Noise-based reward-modulated learning is a novel synaptic plasticity rule.<n>We show that NRL achieves performance comparable to baselines optimized using backpropagation.<n>Results highlight the potential of noise-driven, brain-inspired learning for low-power adaptive systems.
  arXiv  Detail & Related papers  (2025-03-31T11:35:23Z)
• Fast Training of Recurrent Neural Networks with Stationary State Feedbacks [48.22082789438538]
  Recurrent neural networks (RNNs) have recently demonstrated strong performance and faster inference than Transformers.<n>We propose a novel method that replaces BPTT with a fixed gradient feedback mechanism.
  arXiv  Detail & Related papers  (2025-03-29T14:45:52Z)
• Prior Constraints-based Reward Model Training for Aligning Large Language Models [58.33118716810208]
  This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling.
  arXiv  Detail & Related papers  (2024-04-01T07:49:11Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Policy Gradient Bayesian Robust Optimization for Imitation Learning [49.881386773269746]
  We derive a novel policy gradient-style robust optimization approach, PG-BROIL, to balance expected performance and risk. Results suggest PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse.
  arXiv  Detail & Related papers  (2021-06-11T16:49:15Z)
• Robust Learning via Persistency of Excitation [4.674053902991301]
  We show that network training using gradient descent is equivalent to a dynamical system parameter estimation problem. We provide an efficient technique for estimating the corresponding Lipschitz constant using extreme value theory. Our approach also universally increases the adversarial accuracy by 0.1% to 0.3% points in various state-of-the-art adversarially trained models.
  arXiv  Detail & Related papers  (2021-06-03T18:49:05Z)
• Enhancing reinforcement learning by a finite reward response filter with a case study in intelligent structural control [0.0]
  In many reinforcement learning (RL) problems, it takes some time until a taken action by the agent reaches its maximum effect on the environment. This paper introduces an applicable enhanced Q-learning method in which at the beginning of the learning phase, the agent takes a single action. We have applied the developed method to a structural control problem in which the goal of the agent is to reduce the vibrations of a building subjected to earthquake excitations with a specified delay.
  arXiv  Detail & Related papers  (2020-10-25T19:28:35Z)
• Adaptive Gradient Method with Resilience and Momentum [120.83046824742455]
  We propose an Adaptive Gradient Method with Resilience and Momentum (AdaRem) AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient. Our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error.
  arXiv  Detail & Related papers  (2020-10-21T14:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.