Sense Once, Serve Many: Common-Trace Factorized Constrained PPO for Online Sensing-Session Consolidation in Multi-Tenant ISAC Networks

• URL: http://arxiv.org/abs/2608.29256v1
• Date: Sat, 29 Aug 2026 13:20:21 GMT
• Title: Sense Once, Serve Many: Common-Trace Factorized Constrained PPO for Online Sensing-Session Consolidation in Multi-Tenant ISAC Networks
• Abstract summary: We propose Common-Trace Factorized Constrained Proximal Policy Optimization (CT-PPO)<n>CT-PPO achieves the highest mean macro return, exceeding matched Joint-Credit PPO (JC-PPO) by 0.934 and SLA-Aware Greedy by 1.847.<n>A four-way shows that the factorized surrogate alone yields no detectable macro-return gain, whereas adding common-trace reward credit produces the dominant improvement.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrated sensing and communication (ISAC) networks can serve compatible requests through shared sensing sessions, but consolidation couples admission, reuse, profile selection, sensing service-level agreements (SLAs), communication quality of service (QoS), and future commitments. We formulate this problem as a constrained Markov decision process and propose Common-Trace Factorized Constrained Proximal Policy Optimization (CT-PPO). During training, stochastic policy replicas share the same primitive workload trace; leave-one-out discounted Monte Carlo return contrasts provide reward credit to applicable actor factors, while constraint credit remains factor/prefix-specific. Across five training seeds and matched workloads, CT-PPO achieves the highest mean macro return, exceeding matched Joint-Credit PPO (JC-PPO) by 0.934 (95% confidence interval [0.702, 1.164]) and SLA-Aware Greedy by 1.847; versus JC-PPO, it reduces sensing-resource cost by 6.277 and raises accepted requests per created session by 0.0806. A four-way ablation shows that the factorized surrogate alone yields no detectable macro-return gain, whereas adding common-trace reward credit produces the dominant improvement. Without retraining, CT-PPO retains a return advantage at low, nominal, and high arrival loads, with the strongest gain under clustered arrivals. Deployment uses public observations and hard masks; CT-PPO's extra parameters are training-side, its actor footprint matches JC-PPO, and actor-only CPU latency is effectively unchanged.

Related papers

• Mitigating Bias in Low-SNR Financial Reinforcement Learning via Quantum Representations [32.164152251693565]
  Financial market is a typical low signal-to-noise ratio (SNR) setting, which often destabilizes maximum-entropy methods like Soft Actor-Critic (SAC)<n>In this paper, we propose FPQC-SAC, an efficient and bounded riticized Quantum Circuit (PQC) before the actor and critic networks to constrain feature propagation at the representation level.<n> Empirical evaluations on real-world portfolio management tasks demonstrate that FPQC-SAC substantially enhances out-of-sample stability and cumulative returns.
  arXiv  Detail & Related papers  (2026-06-09T05:55:06Z)
• Rethinking the Divergence Regularization in LLM RL [56.952725733148746]
  Reinforcement learning (RL) has become a key component of post-training large language models (LLMs)<n>We propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift.<n>DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights.
  arXiv  Detail & Related papers  (2026-06-08T17:58:23Z)
• When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming [0.0]
  We present an empirical failure-mode study of a compact RLHF pipeline with proximal policy optimization (PPO)<n>We classify matched transitions between checkpoints using the directions of the learned reward, judge scores, and average judge score.<n>A pre-transition logistic model predicts future row-level reward hacking with ROC-AUC 0.821, and row-level analysis finds localized reward hacking that checkpoint averages miss in 3 of 12 settings.
  arXiv  Detail & Related papers  (2026-06-02T06:55:52Z)
• Physics-Informed Causal MDPs for Sequential Constraint Repair in Engineering Simulation Pipelines [0.0]
  We introduce PI-CMDP, a framework for CMDPs whose constraint dependencies form a layered DAG.<n>We instantiate PI-CMDP on constraint repair in engineering simulation pipelines.
  arXiv  Detail & Related papers  (2026-04-20T07:40:15Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization [31.935482701047544]
  We propose Process-Aware Policy Optimization (PAPO) to address two limitations of existing reward designs.<n>PAPO integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization.<n> Experiments across multiple model scales and six benchmarks demonstrate that PAPO consistently outperforms ORM.
  arXiv  Detail & Related papers  (2026-03-27T15:48:13Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• PRISM-FCP: Byzantine-Resilient Federated Conformal Prediction via Partial Sharing [6.066028939131919]
  PRISM-FCP is a Byzantine-resilient Federated Conformal Prediction framework.<n>During training, clients partially share updates by transmitting only $M$ of $D$ parameters per round.<n>During calibration, clients convert nonconformity scores into characterization vectors, compute distance-based maliciousness scores, and downweight or filter suspected Byzantine contributions.
  arXiv  Detail & Related papers  (2026-02-20T18:01:59Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization [15.965340493880701]
  We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework.<n>PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages.<n>On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network.
  arXiv  Detail & Related papers  (2026-01-12T04:04:43Z)
• Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance [52.65461207786633]
  Policy-based Reinforcement Learning from Human Feedback is essential for aligning large language models with human preferences.<n>It requires joint training of an actor and critic with a pretrained, fixed reward model for guidance.<n>We propose textbfDecoupled Value Policy Optimization (DVPO), a lean framework that replaces traditional reward modeling with a pretrained emphglobal value model (GVM)
  arXiv  Detail & Related papers  (2025-02-24T08:11:33Z)
• VinePPO: Refining Credit Assignment in RL Training of LLMs [66.80143024475635]
  We propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates.<n>Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• Monotonic Improvement Guarantees under Non-stationarity for Decentralized PPO [66.5384483339413]
  We present a new monotonic improvement guarantee for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL) We show that a trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training.
  arXiv  Detail & Related papers  (2022-01-31T20:39:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.