Stackelberg Coupling of Online Representation Learning and Reinforcement Learning

• URL: http://arxiv.org/abs/2508.07452v2
• Date: Wed, 01 Oct 2025 15:29:53 GMT
• Title: Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
• Authors: Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen,
• Abstract summary: We introduce SCORER, a framework for value-based RL that views representation and Q-learning as two strategic agents in a hierarchical game.<n>Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by a two-timescale algorithm.
• Score: 45.70357546589222
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Q-learning jointly learns representations and values within monolithic networks, promising beneficial co-adaptation between features and value estimates. Although this architecture has attained substantial success, the coupling between representation and value learning creates instability as representations must constantly adapt to non-stationary value targets, while value estimates depend on these shifting representations. This is compounded by high variance in bootstrapped targets, which causes bias in value estimation in off-policy methods. We introduce Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework for value-based RL that views representation and Q-learning as two strategic agents in a hierarchical game. SCORER models the Q-function as the leader, which commits to its strategy by updating less frequently, while the perception network (encoder) acts as the follower, adapting more frequently to learn representations that minimize Bellman error variance given the leader's committed strategy. Through this division of labor, the Q-function minimizes MSBE while perception minimizes its variance, thereby reducing bias accordingly, with asymmetric updates allowing stable co-adaptation, unlike simultaneous parameter updates in monolithic solutions. Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by a two-timescale algorithm that creates an asymmetric learning dynamic between the two players. Extensive experiments on DQN and its variants demonstrate that gains stem from algorithmic insight rather than model complexity.

Related papers

• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddings [10.36125908359289]
  We present a novel model-based multi-agent reinforcement learning framework.<n>We design a world model trained with variational auto-encoders and augment the model using the state-action learned embedding.<n>By coupling imagined trajectories with SALE-based action values, the agents acquire a richer understanding of how their choices influence collective outcomes.
  arXiv  Detail & Related papers  (2026-02-13T01:57:21Z)
• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning [27.73410730631346]
  We use physics-informed neural networks to approximate HJB-based value functions at scale.<n>This improves gradient fidelity, in turn yielding more accurate values and stronger policy learning.<n>Our results demonstrate that our approach consistently outperforms existing continuous-time baselines and scales to complex multi-agent dynamics.
  arXiv  Detail & Related papers  (2025-09-11T04:12:50Z)
• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning [106.68304931854038]
  Reinforcement learning with verifiable rewards (RLVR) has been widely used for enhancing the reasoning abilities of large language models (LLMs)<n>We conduct a systematic empirical analysis of the entropy-performance exchange mechanism of RLVR across different levels of granularity.<n>Our analysis reveals that, in the rising stage, entropy reduction in negative samples facilitates the learning of effective reasoning patterns.<n>In the plateau stage, learning efficiency strongly correlates with high-entropy tokens present in low-perplexity samples and those located at the end of sequences.
  arXiv  Detail & Related papers  (2025-08-04T10:08:10Z)
• Confidence-driven Gradient Modulation for Multimodal Human Activity Recognition: A Dynamic Contrastive Dual-Path Learning Approach [3.0868241505670198]
  We propose a novel framework called the Dynamic Contrastive Dual-Path Network (D-HAR)<n>The framework comprises three key components. First, a dual-path feature extraction architecture is employed, where ResNet and DenseCDPNet branches collaboratively process multimodal sensor data.<n>Second, a multi-stage contrastive learning mechanism is introduced to achieve progressive alignment from local perception to semantic abstraction.<n>Third, we present a confidence-driven gradient modulation strategy that dynamically monitors and adjusts the learning intensity of each modality branch during backpropagation.
  arXiv  Detail & Related papers  (2025-07-03T17:37:46Z)
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts [79.18608192761512]
  Self-Explainable Models (SEMs) rely on Prototypical Concept Learning (PCL) to enable their visual recognition processes more interpretable.<n>We propose a Few-Shot Prototypical Concept Classification framework that mitigates two key challenges under low-data regimes: parametric imbalance and representation misalignment.<n>Our approach consistently outperforms existing SEMs by a notable margin, with 4.2%-8.7% relative gains in 5-way 5-shot classification.
  arXiv  Detail & Related papers  (2025-06-05T06:39:43Z)
• Dataset Awareness is not Enough: Implementing Sample-level Tail Encouragement in Long-tailed Self-supervised Learning [16.110763554788445]
  We introduce pseudo-labels into self-supervised long-tailed learning, utilizing pseudo-label information to drive a dynamic temperature and re-weighting strategy. We analyze the lack of quantity awareness in the temperature parameter and use re-weighting to compensate for this deficiency, thereby achieving optimal training patterns at the sample level.
  arXiv  Detail & Related papers  (2024-10-30T10:25:22Z)
• A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learning [48.59516337905877]
  Learning a good representation is a crucial challenge for Reinforcement Learning (RL) agents. Recent work has developed theoretical insights into these algorithms. We take a step towards bridging the gap between theory and practice by analyzing an action-conditional self-predictive objective.
  arXiv  Detail & Related papers  (2024-06-04T07:22:12Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [86.99017195607077]
  We address the challenge of sampling and remote estimation for autoregressive Markovian processes in a wireless network with statistically-identical agents.<n>Our goal is to minimize time-average estimation error and/or age of information with decentralized scalable sampling and transmission policies.
  arXiv  Detail & Related papers  (2024-04-04T06:24:11Z)
• Probabilistic Self-supervised Learning via Scoring Rules Minimization [19.347097627898876]
  We propose a novel probabilistic self-supervised learning via Scoring Rule Minimization (ProSMIN) to enhance representation quality and mitigate collapsing representations. Our method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on large-scale datasets.
  arXiv  Detail & Related papers  (2023-09-05T08:48:25Z)
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks [63.24965775030673]
  We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
  arXiv  Detail & Related papers  (2023-02-02T18:27:20Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning [53.17258888552998]
  This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation. We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
  arXiv  Detail & Related papers  (2022-06-01T23:26:51Z)
• Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning [0.0]
  We propose a novel concept of Residual Q-Networks (RQNs) for Multi-Agent Reinforcement Learning (MARL) The RQN learns to transform the individual Q-value trajectories in a way that preserves the Individual-Global-Max criteria (IGM) The proposed method converges faster, with increased stability and shows robust performance in a wider family of environments.
  arXiv  Detail & Related papers  (2022-05-30T16:56:06Z)
• DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction [20.51885543358098]
  We propose DHEN - a deep and hierarchical ensemble architecture that can leverage strengths of heterogeneous interaction modules and learn a hierarchy of the interactions under different orders. Experiments on large-scale dataset from CTR prediction tasks attained 0.27% improvement on the Normalized Entropy of prediction and 1.2x better training throughput than state-of-the-art baseline.
  arXiv  Detail & Related papers  (2022-03-11T21:19:31Z)
• Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning [15.698612710580447]
  We propose a skill learning framework DR-GRL that aims to improve the sample efficiency and policy generalization. In a weakly supervised manner, we propose a Spatial Transform AutoEncoder (STAE) to learn an interpretable and controllable representation. We empirically demonstrate that DR-GRL significantly outperforms the previous methods in sample efficiency and policy generalization.
  arXiv  Detail & Related papers  (2022-02-28T09:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.