Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing

• URL: http://arxiv.org/abs/2607.28814v1
• Date: Thu, 30 Jul 2026 20:16:03 GMT
• Title: Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing
• Abstract summary: In Motivational Interviewing (MI), a client's sustain talk (arguments for the status quo) calls for the counselor to roll with resistance, a move that can fail in two ways.<n>We introduce a two-axis evaluation of counselor responses of Motivational Interviewing Treatment Integrity (MITI) code, Goal Persistence (GP) and Attunement (RA)<n>We ask whether penalizing one failure through preference optimization teaches rolling resistance provokes its opposite.
• Score: 0.6372261626436676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In Motivational Interviewing (MI), a client's sustain talk (arguments for the status quo) calls for the counselor to roll with resistance, a move that can fail in two opposite ways: capitulation (abandoning the change agenda to preserve rapport) or confrontation (arguing or directing, overriding the client's autonomy). We introduce a two-axis evaluation of counselor responses, anchored in the Motivational Interviewing Treatment Integrity (MITI) code, Goal Persistence (GP) and Relational Attunement (RA), yielding a four-quadrant framing in which rolling with resistance is high on both, and we ask whether penalizing one failure through preference optimization teaches rolling with resistance or provokes its opposite. From the expert-annotated AnnoMI corpus we build topic-disjoint Direct Preference Optimization data whose preference sets differ only in which failure is rejected, using on-policy negatives. An automatic judge, validated against AnnoMI's expert labels and rechecked by trained human coders, scores blind pairwise win-rates against each base under a firewall in which disjoint model families generate, label, and judge. Across three aligned instruction models spanning the Qwen and Llama families, penalizing confrontation reliably lowers goal persistence below parity, on every base and in every seed run, a robust cost, whereas the attunement gain is base-dependent, present on two of the three bases but absent on the third. Penalizing capitulation is inert, because these models rarely capitulate on-policy, so the trade is gated by each base's failure profile. A prompt-only control raises attunement without the goal-persistence cost, locating the cost in the optimization rather than in attunement itself.

Related papers

• The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning [6.195031716121927]
  In LURE, an LLM evader positions tasks along each environment's difficulty axis to stay one step ahead of a pursuer that hunts it down through verifiable interaction.<n>The pursuer earns capture-anchored dense process credit, in which monotone verifier progress is group-normalized jointly with the terminal capture under a round-anchored KL that keeps the co-evolution stable.
  arXiv  Detail & Related papers  (2026-08-22T09:30:29Z)
• An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning [0.0]
  Unlearning is evaluated through two objectives: suppress target-specific knowledge and preserve non-target utility.<n>In generative QA, this leaves a third behavior underspecified: when a target-adjacent prompt admits a broader answer without target-specific leakage, the model should answer at that level rather than leak, evade, or refuse.<n>We compare four reward designs that span lexical suppression, anti-refusal shaping, rubric-based broad answering, and an explicit refusal contrast, with and without SFT warm-up.
  arXiv  Detail & Related papers  (2026-08-18T14:04:29Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization [31.547614109943158]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for eliciting long-chain reasoning in large language models.<n>We propose ISPO (Intrinsic Signal Policy Optimization), which combines a sequence-level signal measuring how informative the thinking trajectory is for the final answer, with a token-level directional reward whose hallucinated-certainty hinge penalizes confidently-wrong predictions at critical decision tokens.<n>ISPO consistently outperforms competitive baselines, with the largest gains on the hardest benchmarks where zero-advantage collapse is most frequent, and training-dynamics diagnostics confirm that both failure modes are decreased.
  arXiv  Detail & Related papers  (2026-06-07T20:08:36Z)
• Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics [33.21526869949403]
  We introduce SVR (Support Vectors), a framework that recasts rubrics as max-margin boundary learning over preference data.<n> SVR mines contrastive features from preference pairs into a bank, learns a prompt-conditioned selector together with global weights, and iteratively refines bank through support-pair selection and adversarial probing of hard negatives.<n>OnBench, SVR narrows the gap to human references from 24.1 to 0.3 points and outperforms strong self-rubrics, and the learned bank transfers across judges without retraining.
  arXiv  Detail & Related papers  (2026-06-06T09:55:22Z)
• Dual Optimal: Make Your LLM Peer-like with Dignity [35.58723842086038]
  Current aligned language models exhibit a dual failure mode we term the Evasive Servant.<n>We propose the Dignified Peer framework, which counters servility with anti-sycophancy and trustworthiness.
  arXiv  Detail & Related papers  (2026-04-01T14:48:44Z)
• In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback [38.915062716409686]
  InTRO is a new framework that enables both token-level exploration and self-feedback for accurate and concise reasoning.<n>InTRO consistently outperforms other baselines, raising solution accuracy by up to 20% relative to the base model.<n>Its chains of thought are notably more concise, exhibiting reduced verbosity.
  arXiv  Detail & Related papers  (2025-11-13T01:47:06Z)
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models [50.84995206660551]
  We introduce Conditional advANtage estimatiON (CANON) to amplify the impact of a target metric without presuming its direction.<n>CANON based on entropy consistently outperforms prior methods on both math reasoning and high-complexity logic tasks.
  arXiv  Detail & Related papers  (2025-09-28T16:33:07Z)
• REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective [57.57786477441956]
  We propose an adaptive and semantic optimization problem over the population of responses.<n>Our objective doubles the attack success rate (ASR) on Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.
  arXiv  Detail & Related papers  (2025-02-24T15:34:48Z)
• Making Large Language Models Better Reasoners with Alignment [57.82176656663245]
  Reasoning is a cognitive process of using evidence to reach a sound conclusion. Recent studies reveal that fine-tuning LLMs on data with the chain of thought (COT) reasoning process can significantly enhance their reasoning capabilities. We introduce an textitAlignment Fine-Tuning (AFT) paradigm, which involves three steps.
  arXiv  Detail & Related papers  (2023-09-05T11:32:48Z)
• Alternating Objectives Generates Stronger PGD-Based Adversarial Attacks [78.2700757742992]
  Projected Gradient Descent (PGD) is one of the most effective and conceptually simple algorithms to generate such adversaries. We experimentally verify this assertion on a synthetic-data example and by evaluating our proposed method across 25 different $ell_infty$-robust models and 3 datasets. Our strongest adversarial attack outperforms all of the white-box components of AutoAttack ensemble.
  arXiv  Detail & Related papers  (2022-12-15T17:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.