Mind the Gap: Theory-of-Mind-Grounded Friction for Epistemic Alignment

• URL: http://arxiv.org/abs/2608.30719v1
• Date: Mon, 31 Aug 2026 12:57:29 GMT
• Title: Mind the Gap: Theory-of-Mind-Grounded Friction for Epistemic Alignment
• Abstract summary: We operationalize Theory-of-Mind (ToM) inference as a control signal within Frictive Policy Optimization.<n>At the representation level, ablating the second-order channel reduces misunderstanding recall from $65%$ to $26%$.<n>At the policy level, reward-shaping (FAR) and trust-region (FTR) variants improve intervention F1 and warranted-context calibration over DPO.
• Score: 10.142691415785281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Productive dialogue alignment requires distinguishing \emph{surface coordination} (acknowledgments and smooth task progression) from \emph{epistemic alignment} (convergence of belief states); standard preference-based methods typically optimize response-level preferences without explicitly modeling the latter. We operationalize Theory-of-Mind (ToM) inference as a control signal within Frictive Policy Optimization by extracting, at each referring expression, a four-part belief structure: the speaker's intended referent, the addressee's interpretation, and each participant's model of the other's belief. This makes friction mechanically computable from epistemic-state comparisons, capturing \emph{silent divergence}, where both participants proceed confidently while grounding to different referents. We evaluate the signal at two levels. At the representation level, ablating the second-order channel reduces misunderstanding recall from $65\%$ to $26\%$. At the policy level, reward-shaping (FAR) and trust-region (FTR) variants improve intervention F1 and warranted-context calibration over DPO, with Brier scores independently supporting the calibration gains. Across three training runs, FAR and FTR remain substantially more stable, whereas DPO varies widely and can degrade intervention competence already present in the base policy. Thus, ToM-grounded friction provides a trainable signal for context-sensitive intervention under referential belief divergence.

Related papers

• From Propositional to Perceptual Asymmetry: Extending Frictive Policy Optimization to Asymmetric Partial Information Dialogue [10.142691415785281]
  Frictive Policy Optimization treats friction in collaborative dialogue as a signal essential to common-ground construction.<n>We extend FPO to perceptual asymmetry, where participants hold asymmetric partial information.<n>We find that FPO's friction functional is empirically valid only when evaluated from within each participant's information horizon.
  arXiv  Detail & Related papers  (2026-06-29T23:16:19Z)
• Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization [31.547614109943158]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for eliciting long-chain reasoning in large language models.<n>We propose ISPO (Intrinsic Signal Policy Optimization), which combines a sequence-level signal measuring how informative the thinking trajectory is for the final answer, with a token-level directional reward whose hallucinated-certainty hinge penalizes confidently-wrong predictions at critical decision tokens.<n>ISPO consistently outperforms competitive baselines, with the largest gains on the hardest benchmarks where zero-advantage collapse is most frequent, and training-dynamics diagnostics confirm that both failure modes are decreased.
  arXiv  Detail & Related papers  (2026-06-07T20:08:36Z)
• OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization [58.03221830946145]
  We propose OmniISR, a unified framework that fuses pure CL, pure FL, and hybrid CL-FL training modes.<n>We show that OmniISR consistently improves model performance in both centralized and federated paradigms.
  arXiv  Detail & Related papers  (2026-05-19T04:13:27Z)
• Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline [7.967038299436285]
  We propose a deterministic adjoint matching framework that formulates human preference alignment for flow-based generative models.<n>Building on this perspective, we introduce a truncated adjoint scheme that focuses on the terminal portion of the trajectory.<n>We further generalize the framework beyond standard KL-based regularization, allowing more flexible trade-offs between alignment strength and distributional preservation.
  arXiv  Detail & Related papers  (2026-05-07T17:12:47Z)
• Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration [13.299987551113809]
  Group Cognition Learning (GCL) is a governed collaboration paradigm that applies a two-stage protocol after modality-specific encoding.<n>GCL mitigates modality dominance and coupling, establishing state-of-the-art results across both regression and classification benchmarks.
  arXiv  Detail & Related papers  (2026-05-01T03:19:34Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuning [85.20505958752928]
  Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment.<n>RFT often introduce visual hallucinations like over-optimized details and semantic misalignment.<n>This work preliminarily explores why visual hallucinations arise and how to reduce them.
  arXiv  Detail & Related papers  (2026-02-03T11:49:46Z)
• GRCF: Two-Stage Groupwise Ranking and Calibration Framework for Multimodal Sentiment Analysis [20.77940776708036]
  Pairwise ordinal learning frameworks capture relative order by learning from comparisons.<n>They assign uniform importance to all comparisons, failing to adaptively focus on hard-to-rank samples.<n>We propose a Two-Stage Group-wise Ranking and Framework (GRCF) that adapts the philosophy of Group Relative Policy Optimization.<n>GRCF achieves state-of-the-art performance on core regression benchmarks, while also showing strong generalizability in classification tasks.
  arXiv  Detail & Related papers  (2026-01-14T16:26:44Z)
• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• ETR: Outcome-Guided Elastic Trust Regions for Policy Optimization [6.716883192613149]
  We propose textbfElastic textbfTrust textbfETR, a dynamic mechanism that aligns optimization constraints with signal quality.<n>ETR consistently outperforms GRPO, achieving superior accuracy while effectively mitigating policy entropy degradation.
  arXiv  Detail & Related papers  (2026-01-07T09:19:53Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.