Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment

• URL: http://arxiv.org/abs/2604.25136v1
• Date: Tue, 28 Apr 2026 02:24:51 GMT
• Title: Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment
• Abstract summary: Frictive Policy Optimization (FPO) is a framework for learning language model policies that regulate what to say and when to intervene.<n>We introduce a compact taxonomy of frictive interventions, a structured friction functional that operationalizes multiple alignment failure modes, and a unified family of FPO methods spanning reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions.
• Score: 11.243856942345955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose Frictive Policy Optimization (FPO), a framework for learning language model policies that regulate not only what to say, but when and how to intervene in order to manage epistemic and normative risk. Unlike standard alignment methods that optimize surface-level preference or task utility, FPO treats clarification, verification, challenge, redirection, and refusal as explicit control actions whose purpose is to shape the evolution of belief, commitment, and uncertainty over time. We formalize alignment as a risk-sensitive epistemic control problem in which intervention decisions are selected based on their expected effect on downstream epistemic quality rather than on immediate reward alone. We introduce a compact taxonomy of frictive interventions, a structured friction functional that operationalizes multiple alignment failure modes, and a unified family of FPO methods spanning reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions. We further propose an evaluation framework that measures epistemic competence directly through clarification behavior, calibration, contradiction repair, refusal proportionality, and information efficiency. Together, these results provide a formal and algorithmic foundation for learning agents that are aligned not only in outcome, but in epistemic conduct.

Related papers

• Active Inference as a Convex Markov Decision Process [0.0]
  Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE)<n>We show that, for closed-loop control policies, EFE minimization can be formulated as a Markov decision process (MDP)<n>We argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning.
  arXiv  Detail & Related papers  (2026-07-22T13:52:40Z)
• Behavioural Signatures of Risk-Sensitive Decision-Making in Large Language Models [121.70781851874035]
  Large language models (LLMs) are increasingly used in decision support.<n>It is important to understand whether their choices under uncertainty exhibit stable and interpretable behavioural regularities.<n>Here we examine this question using a controlled multi-model framework based on no-limit Texas Hold'em.
  arXiv  Detail & Related papers  (2026-07-11T10:33:39Z)
• Structural Distinguishability of Static and Adaptive Policy Regimes in Agent-Based Regulatory Simulation [0.0]
  This paper contributes a controlled simulation benchmark rather than a new general framework.<n>We evaluate naive fixed policies, tracking-aware fixed policies, and three adaptive controllers: setpoint, safety-margin, and one-sided control.<n>The contribution is methodological: scalar indicators, cap-relative symbolic diagnostics, trajectory motifs, and visual inspection jointly reveal how regulatory conclusions can differ even when average outcomes appear similar.
  arXiv  Detail & Related papers  (2026-06-15T10:06:32Z)
• Set-Valued Policy Learning [17.33185980499072]
  We propose a set-valued policy learning paradigm for the multiple-treatment setting, in which policies output a set of plausible treatments rather than a single recommendation.<n>We extend the learning-to-defer framework to multiple treatments via a novel textitgreatest Lower Bound method, and introduce textitconformal policy learning, which bridges the gap between unobserved ground-truth optimal treatments and estimated optimal treatment rules.
  arXiv  Detail & Related papers  (2026-05-19T13:24:26Z)
• OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents [74.20327254615854]
  Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks.<n>Existing inference-time adaptation methods for LLM agents mainly rely on prompting or retrieval.<n>We propose OLIVIA, an inference-time action adaptation framework for ReAct-style agents.
  arXiv  Detail & Related papers  (2026-05-11T19:28:20Z)
• What Do We Care About in Bandits with Noncompliance? BRACE: Bandits with Recommendations, Abstention, and Certified Effects [0.0]
  We show that recommendation welfare can strictly exceed every learner-measurable treatment policy when downstream actors use private information.<n>For finite-context square-IV problems we propose BRACE, a parameter-free phase-doubling algorithm that performs IV inversion only after matrix certification.<n>We complement the theory with a finite-context empirical benchmark spanning direct control, mediated present-versus-future tradeoffs, weak identification, homogeneity failure, and rectangular overidentification.
  arXiv  Detail & Related papers  (2026-03-10T11:40:42Z)
• When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering [10.01278648231868]
  Policy steering is an emerging way to adapt robot behaviors at deployment-time.<n> Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities.<n>We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility.
  arXiv  Detail & Related papers  (2026-02-25T23:23:22Z)
• Admissibility Alignment [0.0]
  We present MAP-AI, a new control-plane system architecture for aligned decision-making under uncertainty.<n>It enforces alignment through Monte Carlo estimation of outcome distributions and admissibility-controlled policy selection.<n>We show how alignment evaluation can be integrated into decision-making itself, yielding an admissibility-controlled action selection mechanism.
  arXiv  Detail & Related papers  (2026-01-05T05:58:19Z)
• Transfer Learning for Classification under Decision Rule Drift with Application to Optimal Individualized Treatment Rule Estimation [50.34670342434884]
  We propose a novel methodology for modeling posterior drift through Bayes decision rules.<n>Under mild regularity conditions, we establish the consistency of our estimators and derive the risk bounds.<n>We illustrate the broad applicability of our method by adapting it to the estimation of optimal individualized treatment rules.
  arXiv  Detail & Related papers  (2025-08-28T16:03:06Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Assessing the Impact of Context Inference Error and Partial Observability on RL Methods for Just-In-Time Adaptive Interventions [12.762365585427377]
  Just-in-Time Adaptive Interventions (JITAIs) are a class of personalized health interventions developed within the behavioral science community. JITAIs aim to provide the right type and amount of support by iteratively selecting a sequence of intervention options from a pre-defined set of components. We study the effect of context inference error and partial observability on the ability to learn effective policies.
  arXiv  Detail & Related papers  (2023-05-17T02:46:37Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• Optimal Inspection and Maintenance Planning for Deteriorating Structural Components through Dynamic Bayesian Networks and Markov Decision Processes [0.0]
  Partially Observable Markov Decision Processes (POMDPs) provide a mathematical methodology for optimal control under uncertain action outcomes and observations. We provide the formulation for developing both infinite and finite horizon POMDPs in a structural reliability context. Results show that POMDPs achieve substantially lower costs as compared to their counterparts, even for traditional problem settings.
  arXiv  Detail & Related papers  (2020-09-09T20:03:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.