The Thinking Therapist: Training Large Language Models to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization

• URL: http://arxiv.org/abs/2509.09712v2
• Date: Sat, 20 Sep 2025 21:31:47 GMT
• Title: The Thinking Therapist: Training Large Language Models to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization
• Abstract summary: Acceptance and Commitment Therapy (ACT) is a cognitive behavioral therapy with emerging evidence of efficacy in several psychiatric conditions.<n>This study investigates the impact of post-training methodology and explicit reasoning on the ability of a small open-weight large language model (LLM) to deliver ACT.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Acceptance and Commitment Therapy (ACT) is a third-wave cognitive behavioral therapy with emerging evidence of efficacy in several psychiatric conditions. This study investigates the impact of post-training methodology and explicit reasoning on the ability of a small open-weight large language model (LLM) to deliver ACT. Using synthetic ACT transcripts generated by Mistral-Large, we trained Llama-3.2-3b-Instruct with two distinct approaches, supervised fine-tuning (SFT) and odds ratio policy optimization (ORPO), each with and without an explicit chain-of-thought (COT) reasoning step. Performance was evaluated by comparing these four post-trained variants against the base Instruct model. These models were benchmarked in simulated therapy sessions, with performance quantitatively assessed on the ACT Fidelity Measure (ACT-FM) and the Therapist Empathy Scale (TES) by an LLM judge that had been fine-tuned on human evaluations. Our findings demonstrate that the ORPO-trained models significantly outperformed both their SFT and Instruct counterparts on ACT fidelity ($\chi^2(5) = 185.15, p < .001$) and therapeutic empathy ($\chi^2(5) = 140.37, p < .001$). The effect of COT was conditional as it provided a significant benefit to SFT models, improving ACT-FM scores by an average of 2.68 points ($p < .001$), while offering no discernible advantage to the superior ORPO or instruct-tuned variants. We posit that the superiority of ORPO stems from its ability to learn the therapeutic `process' over imitating `content,' a key aspect of ACT, while COT acts as a necessary scaffold for models trained only via imitation. This study establishes that preference-aligned policy optimization can effectively instill ACT competencies in small LLMs, and that the utility of explicit reasoning is highly dependent on the underlying training paradigm.

Related papers

• Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars [0.45671221781968324]
  Training psychotherapists in Acceptance and Commitment Therapy (ACT) requires repeated practice with meaningful feedback.<n>We introduce a system designed to support ACT-oriented psychotherapy training through spoken dialogue with an embodied virtual patient.
  arXiv  Detail & Related papers  (2026-06-16T11:02:00Z)
• DeepImagine: Learning Biomedical Reasoning via Successive Counterfactual Imagining [6.221068729338845]
  We propose a framework for teaching LLMs biomedical reasoning through successive counterfactual imagining.<n>We construct both natural and approximate counterfactual pairs from real clinical trials with reported outcomes.<n>We show that DeepImagine consistently improves over untuned language models and traditional correlational baselines.
  arXiv  Detail & Related papers  (2026-04-24T22:48:10Z)
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training [69.32453275232662]
  We show that a learned critic can inject estimation noise that exceeds the state signal it captures, increasing rather than reducing advantage variance.<n>We propose Explained Variance Policy Optimization (EVPO), which monitors batch-level EV at each training step and adaptively switches between critic-based and batch-mean advantage estimation.
  arXiv  Detail & Related papers  (2026-04-21T14:07:39Z)
• Agentic Critical Training [58.33938417298441]
  Training large language models (LLMs) as autonomous agents often begins with imitation learning.<n>We propose Agentic Critical Training (ACT), a reinforcement learning paradigm that trains agents to identify the better action among alternatives.<n>ACT consistently improves agent performance when combined with different post-training methods.
  arXiv  Detail & Related papers  (2026-03-09T17:58:56Z)
• OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Prediction [2.904892426557913]
  Large language models (LLMs) have shown strong performance in biomedical NLP.<n>We present a unified, multi-task learning framework that aligns autoregressive LLMs with clinical reasoning for outcome prediction.<n>Our findings underscore the importance of reasoning-aware alignment in multi-task clinical modeling.
  arXiv  Detail & Related papers  (2025-10-20T13:35:12Z)
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models [50.84995206660551]
  We introduce Conditional advANtage estimatiON (CANON) to amplify the impact of a target metric without presuming its direction.<n>CANON based on entropy consistently outperforms prior methods on both math reasoning and high-complexity logic tasks.
  arXiv  Detail & Related papers  (2025-09-28T16:33:07Z)
• HEFT: A Coarse-to-Fine Hierarchy for Enhancing the Efficiency and Accuracy of Language Model Reasoning [0.0]
  HEFT is a novel hierarchical adaptation strategy that composes two distinct PEFT methods in a coarse-to-fine manner.<n>A model fine-tuned for only three epochs with our HEFT strategy achieves an accuracy of 85.17%, exceeding the performance of models trained for 20 epochs.
  arXiv  Detail & Related papers  (2025-09-11T19:06:46Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• Evaluating AI Alignment in Eleven LLMs through Output-Based Analysis and Human Benchmarking [0.0]
  Large language models (LLMs) are increasingly used in psychological research and practice, yet traditional benchmarks reveal little about the values they express in real interaction.<n>We introduce PAPERS, output-based evaluation of the values LLMs express.
  arXiv  Detail & Related papers  (2025-06-14T20:14:02Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• Teaching LLMs to Refine with Tools [68.23479664749271]
  Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement.<n>We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs.
  arXiv  Detail & Related papers  (2024-12-22T05:43:50Z)
• Fine-Tuning Open-Weight Language Models to Deliver Cognitive Behavioral Therapy for Depression: A Feasibility Study [0.0]
  Cognitive Behavioral Therapy (CBT) is an evidence-based treatment for Major Depressive Disorder.<n>There exist significant barriers to individuals accessing CBT, including cost, scarcity of therapists and stigma.<n>This study explores the feasibility of fine-tuning small open language models (LLMs) to deliver CBT for depression.
  arXiv  Detail & Related papers  (2024-11-29T20:48:08Z)
• Poor-Supervised Evaluation for SuperLLM via Mutual Consistency [20.138831477848615]
  We propose the PoEM framework to conduct evaluation without accurate labels. We first prove that the capability of a model can be equivalently assessed by the consistency between it and certain reference model. To alleviate the insufficiencies of the conditions in reality, we introduce an algorithm that treats humans (when available) and the models under evaluation as reference models.
  arXiv  Detail & Related papers  (2024-08-25T06:49:03Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)
• Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines [4.482833270578723]
  Large Language Models (LLMs) can significantly improve Clinical Decision Support (CDS) We develop three distinct methods for incorporating Clinical Practice Guidelines (CPGs) into LLMs. We focus on CDS for COVID-19 outpatient treatment as the case study.
  arXiv  Detail & Related papers  (2024-01-20T05:10:46Z)
• Individualized Policy Evaluation and Learning under Clustered Network Interference [3.8601741392210434]
  We consider the problem of evaluating and learning an optimal individualized treatment rule (ITR) under clustered network interference.<n>We propose an estimator that can be used to evaluate the empirical performance of an ITR.<n>We derive the finite-sample regret bound for a learned ITR, showing that the use of our efficient evaluation estimator leads to the improved performance of learned policies.
  arXiv  Detail & Related papers  (2023-11-04T17:58:24Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)
• Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection [73.31406286956535]
  We introduce the Ladder-of-Thought (LoT) for the stance detection task. LoT directs the small LMs to assimilate high-quality external knowledge, refining the intermediate rationales produced. Our empirical evaluations underscore LoT's efficacy, marking a 16% improvement over GPT-3.5 and a 10% enhancement compared to GPT-3.5 with CoT on stance detection task.
  arXiv  Detail & Related papers  (2023-08-31T14:31:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.