Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing

• URL: http://arxiv.org/abs/2603.15647v1
• Date: Wed, 04 Mar 2026 08:00:16 GMT
• Title: Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing
• Abstract summary: We introduce the Consensus Clustering LinUCB Bandit (CCLUB), a unified framework for adaptive social alignment via system-prompt routing.<n>CCLUB employs a conservative consensus clustering mechanism, effectively preventing unsafe generalization across semantically proximal but risk-divergent contexts.<n>Our theoretical analysis yields a sublinear regret guarantee, demonstrating near-optimal performance of CCLUB.
• Score: 37.86964394689924
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are typically governed by post-training alignment (e.g., RLHF or DPO), which yields a largely static policy during deployment and inference. However, real-world safety is a full-lifecycle problem: static defenses degrade against evolving jailbreak behaviors, and fixed weights cannot adapt to pluralistic, time-varying safety norms. This motivates inference-time governance that steers behavior without costly retraining. To address this, we introduce the Consensus Clustering LinUCB Bandit (CCLUB), a unified framework for adaptive social alignment via system-prompt routing. CCLUB employs a conservative consensus clustering mechanism: it pools data only within the intersection of utility and safety similarity graphs, effectively preventing unsafe generalization across semantically proximal but risk-divergent contexts. Our theoretical analysis yields a sublinear regret guarantee, demonstrating near-optimal performance of CCLUB. Extensive experiments validate that CCLUB outperforms strong baselines, achieving a 10.98% improvement in cumulative reward and a 14.42% reduction in the average suboptimality gap.

Related papers

• UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma [32.401066562991744]
  Unbounded Positive Asymmetric Optimization (UP) is a universal and plug-and-play objective.<n>UP restructures the optimization process by anchoring the policy to its current state via the stop-gradient operator.<n>UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms.
  arXiv  Detail & Related papers  (2026-07-08T04:21:42Z)
• BSO: Safety Alignment Is Density Ratio Matching [14.960171907231663]
  We show that the likelihood of the optimal safe policy admits a closed-form decomposition that reduces safety alignment to a density ratio matching problem.<n>Minimizing Bregman divergences between the data and model ratios yields Bregman Safety Optimization (BSO), a family of single-stage loss functions, that provably recover the optimal safe policy.<n> Experiments across safety alignment benchmarks show that BSO consistently improves the safety-helpfulness trade-off.
  arXiv  Detail & Related papers  (2026-05-12T16:19:42Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding [59.16244104797919]
  We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**.<n>DARC frames response selection as distributionally robust, risk-sensitive decision making.<n>Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality.
  arXiv  Detail & Related papers  (2026-03-09T09:21:29Z)
• Unifying Stable Optimization and Reference Regularization in RLHF [64.16830602324345]
  This paper introduces a unified regularization approach that balances objectives of preventing reward hacking and maintaining stable policy updates.<n>Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity.
  arXiv  Detail & Related papers  (2026-02-12T03:31:19Z)
• Robust Policy Optimization to Prevent Catastrophic Forgetting [29.514746370429965]
  Large language models are commonly trained through multi-stage post-training.<n>Small downstream updates can compromise earlier learned behaviors.<n>This suggests standard RLHF objectives do not guarantee robustness to future adaptation.
  arXiv  Detail & Related papers  (2026-02-09T15:50:05Z)
• Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment [45.772620696660034]
  Safety Optimal Transport (SOT) is a novel framework that reframes safe fine-tuning from an instance-level filtering challenge to a distribution-level alignment task grounded in Optimal Transport (OT)<n>SOT prioritizes sample importance by actively pulling the downstream distribution towards a trusted safe anchor while simultaneously pushing it away from a general harmful reference.<n>Experiments across diverse model families and domains demonstrate that SOT significantly enhances model safety while maintaining competitive downstream performance.
  arXiv  Detail & Related papers  (2026-01-12T04:48:02Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)
• ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty [28.291179179647795]
  We propose textbfAdaptive Rank Representation (AdaRL), a bi-level optimization framework that improves robustness.<n>At the lower level, AdaRL performs policy optimization under fixed-rank constraints with dynamics sampled from a Wasserstein ball around a centroid model.<n>At the upper level, it adaptively adjusts the rank to balance the bias--variance trade-off, projecting policy parameters onto a low-rank manifold.
  arXiv  Detail & Related papers  (2025-10-13T20:05:34Z)
• SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning [26.554847852013737]
  SoNIC is the first algorithm that integrates adaptive conformal inference and constrained reinforcement learning.<n>Our method achieves a success rate of 96.93%, which is 11.67% higher than the previous state-of-the-art RL method.<n>Our experiments demonstrate that the system can generate robust and socially polite decision-making when interacting with both sparse and dense crowds.
  arXiv  Detail & Related papers  (2024-07-24T17:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.