SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control

• URL: http://arxiv.org/abs/2605.00787v1
• Date: Fri, 01 May 2026 17:09:58 GMT
• Title: SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
• Abstract summary: We propose a geometry-aware RL algorithm that explicitly incorporates value-based similarity into the policy update.<n>State-Action Value Geometry Optimization (SAVGO) learns a joint state-action embedding space in which pairs with similar action-value estimates exhibit high cosine similarity.<n>This learned geometry enables the generation of a similarity kernel over candidate actions sampled at each update, allowing policy improvement to be guided directly toward higher-value regions.
• Score: 1.5469452301122175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While representation and similarity learning have improved the sample efficiency of Reinforcement Learning (RL), they are rarely used to shape policy updates directly in the action space. To bridge this gap, a geometry-aware RL algorithm that explicitly incorporates value-based similarity into the policy update, State-Action Value Geometry Optimization (SAVGO), is proposed. In detail, SAVGO learns a joint state-action embedding space in which pairs with similar action-value estimates exhibit high cosine similarity, while dissimilar pairs are mapped to distinct directions. This learned geometry enables the generation of a similarity kernel over candidate actions sampled at each update, allowing policy improvement to be guided directly toward higher-value regions beyond local gradient-based updates. As a result, representation learning, value estimation, and policy optimization are unified within a single geometry-consistent objective, while preserving the scalability of off-policy actor-critic training. The proposed method is evaluated on standard MuJoCo continuous-control benchmarks, demonstrating improvements over strong baselines on challenging high-dimensional tasks. Ablation studies are done to analyze the contributions of value-geometry learning and similarity-based policy updates.

Related papers

• SR-OPSD: Self-Referenced On-Policy Self-Distillation [18.590739659841926]
  On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized.<n>We propose the emphSelf-Referenced On-Policy Self-Distillation (SR-OPSD) to achieve state-of-the-art or competitive performance across various settings.
  arXiv  Detail & Related papers  (2026-08-10T15:40:07Z)
• Actor-Critic with Active Importance Sampling [61.41272490044518]
  This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm.<n>AISAC is an extension of the Actor-Critic framework for reducing variance in policy gradient estimation.<n> Experiments show improved learning speed, sample efficiency, and training compared to standard Actor-Critic methods.
  arXiv  Detail & Related papers  (2026-05-08T01:21:32Z)
• A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study [12.940253083606066]
  Reinforcement learning algorithms have been widely used in dynamic and control systems.<n>This study extends that method into a framework which provides a multi-perspective view of the learning dynamics.<n>The framework is applied to compare several ADHDP variants and shows how training stabilizers and target updates change the optimization landscape.
  arXiv  Detail & Related papers  (2026-03-15T20:47:30Z)
• Learning Policy Representations for Steerable Behavior Synthesis [80.4542176039074]
  Given a Markov decision process (MDP), we seek to learn representations for a range of policies to facilitate behavior steering at test time.<n>We show that these representations can be approximated uniformly for a range of policies using a set-based architecture.<n>We use variational generative approach to induce a smooth latent space, and further shape it with contrastive learning so that latent distances align with differences in value functions.
  arXiv  Detail & Related papers  (2026-01-29T21:52:06Z)
• Relative Entropy Pathwise Policy Optimization [66.03329137921949]
  We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories.<n>We show how to combine policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning.
  arXiv  Detail & Related papers  (2025-07-15T06:24:07Z)
• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Model-based trajectory stitching for improved behavioural cloning and its applications [7.462336024223669]
  Trajectory Stitching (TS) generates new trajectories by stitching' pairs of states that were disconnected in the original data. We demonstrate that the iterative process of replacing old trajectories with new ones incrementally improves the underlying behavioural policy.
  arXiv  Detail & Related papers  (2022-12-08T14:18:04Z)
• A Parametric Class of Approximate Gradient Updates for Policy Optimization [47.69337420768319]
  We develop a unified perspective that re-expresses the underlying updates in terms of a limited choice of gradient form and scaling function. We obtain novel yet well motivated updates that generalize existing algorithms in a way that can deliver benefits both in terms of convergence speed and final result quality.
  arXiv  Detail & Related papers  (2022-06-17T01:28:38Z)
• Decoupling Value and Policy for Generalization in Reinforcement Learning [20.08992844616678]
  We argue that more information is needed to accurately estimate the value function than to learn the optimal policy. We propose two approaches which are combined to create IDAAC: Invariant Decoupled Advantage Actor-Critic. IDAAC shows good generalization to unseen environments, achieving a new state-of-the-art on the Procgen benchmark and outperforming popular methods on DeepMind Control tasks with distractors.
  arXiv  Detail & Related papers  (2021-02-20T12:40:11Z)
• Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
  Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
  arXiv  Detail & Related papers  (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.