LLM-as-a-Supervisor: Mistaken Therapeutic Behaviors Trigger Targeted Supervisory Feedback

• URL: http://arxiv.org/abs/2508.09042v1
• Date: Tue, 12 Aug 2025 16:03:36 GMT
• Title: LLM-as-a-Supervisor: Mistaken Therapeutic Behaviors Trigger Targeted Supervisory Feedback
• Abstract summary: Large language models (LLMs) hold significant promise in psychotherapy, but their direct application in patient-facing scenarios raises ethical and safety concerns.<n>This work shifts towards developing an LLM as a supervisor to train real therapists.
• Score: 27.944400312023404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although large language models (LLMs) hold significant promise in psychotherapy, their direct application in patient-facing scenarios raises ethical and safety concerns. Therefore, this work shifts towards developing an LLM as a supervisor to train real therapists. In addition to the privacy of clinical therapist training data, a fundamental contradiction complicates the training of therapeutic behaviors: clear feedback standards are necessary to ensure a controlled training system, yet there is no absolute "gold standard" for appropriate therapeutic behaviors in practice. In contrast, many common therapeutic mistakes are universal and identifiable, making them effective triggers for targeted feedback that can serve as clearer evidence. Motivated by this, we create a novel therapist-training paradigm: (1) guidelines for mistaken behaviors and targeted correction strategies are first established as standards; (2) a human-in-the-loop dialogue-feedback dataset is then constructed, where a mistake-prone agent intentionally makes standard mistakes during interviews naturally, and a supervisor agent locates and identifies mistakes and provides targeted feedback; (3) after fine-tuning on this dataset, the final supervisor model is provided for real therapist training. The detailed experimental results of automated, human and downstream assessments demonstrate that models fine-tuned on our dataset MATE, can provide high-quality feedback according to the clinical guideline, showing significant potential for the therapist training scenario.

Related papers

• MyMentorLLM: A psychotherapy GenAI environment with multimodal voice/text patients, trainees and experts for deliberate practice [41.99844472131922]
  MyMentorLLM is a voice- and text-based simulation environment for deliberate practice.<n>It generates 2,100 complete Cognitive Behavioural Therapy (CBT) training sessions.<n>Simulated patients expressed disorder-congruent emotional profiles.<n>Supervisors' feedback led to better diagnoses in simulated psychotherapists.
  arXiv  Detail & Related papers  (2026-07-28T12:50:11Z)
• Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support [22.031194303408792]
  We introduce a framework that formulates therapeutic response generation as a decision-refinement problem driven by human-aligned evaluation.<n>In Stage I, we introduce TheraJudge, an open-source therapeutic evaluator trained via preference-based optimization on human-annotated data.<n>In Stage II, we introduce TheraAgent, which operationalizes TheraJudge's evaluations through a coordinated refinement process.
  arXiv  Detail & Related papers  (2026-06-29T20:22:25Z)
• Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars [0.45671221781968324]
  Training psychotherapists in Acceptance and Commitment Therapy (ACT) requires repeated practice with meaningful feedback.<n>We introduce a system designed to support ACT-oriented psychotherapy training through spoken dialogue with an embodied virtual patient.
  arXiv  Detail & Related papers  (2026-06-16T11:02:00Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback [52.053200111780626]
  Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident skill acquisition.<n>We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training.<n>Our method uses multi-agent and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria.
  arXiv  Detail & Related papers  (2026-05-25T05:31:44Z)
• Virtual Speech Therapist: A Clinician-in-the-Loop AI Speech Therapy Agent for Personalized and Supervised Therapy [8.877732578001098]
  This paper develops Virtual Speech Therapist (VST), an intelligent agent-based platform that streamlines stuttering assessment and delivers customized therapy planning through automated and adaptive AI-driven.<n>VST integrates state-of-the-art deep learning-based stuttering classification, and multi-agent large language model (LLM) reasoning to support evidence-based clinical decision-making.
  arXiv  Detail & Related papers  (2026-05-01T21:14:45Z)
• PsyProbe: Proactive and Interpretable Dialogue through User State Modeling for Exploratory Counseling [14.53071190134928]
  PsyProbe is a dialogue system designed for the exploration phase of counseling.<n>It tracks user psychological states through the PPPPPI framework.<n>It generates contextually appropriate, proactive questions.
  arXiv  Detail & Related papers  (2026-01-27T01:59:41Z)
• Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models [48.95516224614331]
  We introduce MedGaze-Bench, the first benchmark leveraging clinician gaze as a Cognitive Cursor to assess intent understanding across surgery, emergency simulation, and diagnostic interpretation.<n>Our benchmark addresses three fundamental challenges: visual homogeneity of anatomical structures, strict temporal-causal dependencies in clinical, and implicit adherence to safety protocols.
  arXiv  Detail & Related papers  (2026-01-11T02:20:40Z)
• Trustworthy AI Psychotherapy: Multi-Agent LLM Workflow for Counseling and Explainable Mental Disorder Diagnosis [11.025486717604972]
  DSM5AgentFlow is the first LLM-based agent workflow designed to autonomously generate DSM-5 Level-1 diagnostic questionnaires.<n>By simulating therapist-client dialogues with specific client profiles, the framework delivers transparent, step-by-step disorder predictions.<n>This workflow serves as a complementary tool for mental health diagnosis, ensuring adherence to ethical and legal standards.
  arXiv  Detail & Related papers  (2025-08-15T11:08:32Z)
• Reframe Your Life Story: Interactive Narrative Therapist and Innovative Moment Assessment with Large Language Models [92.93521294357058]
  Narrative therapy helps individuals transform problematic life stories into empowering alternatives.<n>Current approaches lack realism in specialized psychotherapy and fail to capture therapeutic progression over time.<n>Int (Interactive Narrative Therapist) simulates expert narrative therapists by planning therapeutic stages, guiding reflection levels, and generating contextually appropriate expert-like responses.
  arXiv  Detail & Related papers  (2025-07-27T11:52:09Z)
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models [52.2001050216955]
  Existing methods aim to enhance the performance of Medical Vision Language Model (MedVLM) by adjusting model structure, fine-tuning with high-quality data, or through preference fine-tuning.<n>We propose an expert-in-the-loop framework named Expert-Controlled-Free Guidance (Expert-CFG) to align MedVLM with clinical expertise without additional training.
  arXiv  Detail & Related papers  (2025-07-12T09:03:30Z)
• A Risk Taxonomy for Evaluating AI-Powered Psychotherapy Agents [10.405048273969085]
  We introduce a novel risk taxonomy specifically designed for the systematic evaluation of conversational AI psychotherapists.<n>We discuss two use cases in detail: monitoring cognitive model-based risk factors during a counseling conversation to detect unsafe deviations, and in automated benchmarking of AI psychotherapists with simulated patients.
  arXiv  Detail & Related papers  (2025-05-21T05:01:39Z)
• Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback [51.26493826461026]
  We propose Psi-Arena, an interactive framework for comprehensive assessment and optimization of large language models (LLMs)<n>Arena features realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients.<n>Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives.
  arXiv  Detail & Related papers  (2025-05-06T08:22:51Z)
• Position: Beyond Assistance - Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Care [9.30684296057698]
  This position paper argues for a shift in how Large Language Models (LLMs) are integrated into the mental health care domain.<n>We advocate for their role as co-creators rather than mere assistive tools.
  arXiv  Detail & Related papers  (2025-02-21T21:41:20Z)
• Multi-Modal Self-Supervised Learning for Surgical Feedback Effectiveness Assessment [66.6041949490137]
  We propose a method that integrates information from transcribed verbal feedback and corresponding surgical video to predict feedback effectiveness. Our findings show that both transcribed feedback and surgical video are individually predictive of trainee behavior changes. Our results demonstrate the potential of multi-modal learning to advance the automated assessment of surgical feedback.
  arXiv  Detail & Related papers  (2024-11-17T00:13:00Z)
• From Barriers to Tactics: A Behavioral Science-Informed Agentic Workflow for Personalized Nutrition Coaching [0.2808786262095875]
  This paper introduces a novel LLM-powered agentic workflow designed to provide personalized nutrition coaching. A specialized LLM agent intentionally probes for and identifies the root cause of a patient's dietary struggles. A separate LLM agent delivers tailored tactics designed to overcome those specific barriers with patient context.
  arXiv  Detail & Related papers  (2024-10-17T21:35:07Z)
• An Active Inference Strategy for Prompting Reliable Responses from Large Language Models in Medical Practice [0.0]
  Large Language Models (LLMs) are non-deterministic, may provide incorrect or harmful responses, and cannot be regulated to assure quality control. Our proposed framework refines LLM responses by restricting their primary knowledge base to domain-specific datasets containing validated medical information. We conducted a validation study where expert cognitive behaviour therapy for insomnia therapists evaluated responses from the LLM in a blind format.
  arXiv  Detail & Related papers  (2024-07-23T05:00:18Z)
• Automated Fidelity Assessment for Strategy Training in Inpatient Rehabilitation using Natural Language Processing [53.096237570992294]
  Strategy training is a rehabilitation approach that teaches skills to reduce disability among those with cognitive impairments following a stroke. Standardized fidelity assessment is used to measure adherence to treatment principles. We developed a rule-based NLP algorithm, a long-short term memory (LSTM) model, and a bidirectional encoder representation from transformers (BERT) model for this task.
  arXiv  Detail & Related papers  (2022-09-14T15:33:30Z)
• MIA-Prognosis: A Deep Learning Framework to Predict Therapy Response [58.0291320452122]
  This paper aims at a unified deep learning approach to predict patient prognosis and therapy response. We formalize the prognosis modeling as a multi-modal asynchronous time series classification task. Our predictive model could further stratify low-risk and high-risk patients in terms of long-term survival.
  arXiv  Detail & Related papers  (2020-10-08T15:30:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.