PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments

• URL: http://arxiv.org/abs/2601.20330v1
• Date: Wed, 28 Jan 2026 07:48:39 GMT
• Title: PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments
• Abstract summary: We argue that current evaluation paradigms suffer from an unanchored defect, leading to two forms of instability.<n>We introduce Ps, a unified framework that calibrates the therapeutic competence of LLMs via trajectory-anchored tournaments.
• Score: 41.84590571482057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While large language models show promise in mental healthcare, evaluating their therapeutic competence remains challenging due to the unstructured and longitudinal nature of counseling. We argue that current evaluation paradigms suffer from an unanchored defect, leading to two forms of instability: process drift, where unsteered client simulation wanders away from specific counseling goals, and standard drift, where static pointwise scoring lacks the stability for reliable judgment. To address this, we introduce Ps, a unified framework that calibrates the therapeutic competence of LLMs via trajectory-anchored tournaments. We first anchor the interaction trajectory in simulation, where clients precisely control the fluid consultation process to probe multifaceted capabilities. We then anchor the battle trajectory in judgments through an efficient Swiss-system tournament, utilizing dynamic pairwise battles to yield robust Elo ratings. Beyond ranking, we demonstrate that tournament trajectories can be transformed into credible reward signals, enabling on-policy reinforcement learning to enhance LLMs' performance. Extensive experiments validate the effectiveness of PsychePass and its strong consistency with human expert judgments.

Related papers

• CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents [66.9575676266637]
  Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments.<n>In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution.<n>We develop CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization.
  arXiv  Detail & Related papers  (2026-08-31T01:59:51Z)
• PhysReflect-VLA: Physical Feasibility and Self-Reflective Regulation for Reliable Vision-Language-Action Policies [29.471598488144277]
  Long-horizon robotic manipulation is highly sensitive to physically infeasible transitions, contact-induced disturbances, and the lack of effective self-correction during execution.<n>We present PhysReflect-VLA, a plug-and-play execution-time reliability framework that augments VLA policies with physical feasibility evaluation and structured self-reflection in a closed-loop control pipeline.
  arXiv  Detail & Related papers  (2026-06-25T15:18:10Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges [0.519554837386174]
  We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge.<n>We find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge.<n>These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence.
  arXiv  Detail & Related papers  (2026-06-03T19:37:23Z)
• When Clients Stop Following: A Cognitive Conceptualization Diagram-driven Framework for Strategic Counseling [7.733715213738371]
  Existing benchmarks rely heavily on highly cooperative simulated clients.<n>These clients often rapidly shift from resistance to compliance after only a few turns.<n>We propose a Cognitive Behavioral Therapy (CBT)-grounded resistance-aware framework.
  arXiv  Detail & Related papers  (2026-06-03T03:09:10Z)
• EMPA: Evaluating Persona-Aligned Empathy as a Process [2.6558732808569805]
  EMPA is a process-oriented framework that evaluates persona-aligned support as sustained intervention rather than isolated replies.<n>It distills real interactions into controllable, psychologically grounded scenarios.<n>It scores trajectories in a latent psychological space by directional alignment, cumulative impact, and stability.
  arXiv  Detail & Related papers  (2026-02-28T09:04:06Z)
• CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models [19.027335814014528]
  We present CARE, an LLM-based framework to automatically predict multi-dimensional alliance scores and generate interpretable rationales from counseling transcripts.<n> CARE is built on the CounselingWAI dataset and enriched with 9,516 expert-curated rationales.<n>Experiments show that CARE outperforms leading LLMs and substantially reduces the gap between counselor evaluations and client-perceived alliance.
  arXiv  Detail & Related papers  (2026-02-24T07:52:56Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning [29.778703252962092]
  Reinforcement Learning (RL) has emerged as a powerful paradigm for advancing Large Language Models (LLMs)<n>We develop a novel test-time reward mechanism that operates without external supervision.
  arXiv  Detail & Related papers  (2025-10-20T07:53:51Z)
• Confidence as a Reward: Transforming LLMs into Reward Models [54.98336080630691]
  Confidence-as-a-Reward (CRew) is a training-free method that utilizes token-level confidence in the model's final answers as a proxy for reward.<n>We show that CRew outperforms existing training-free reward approaches on the MATH500 and RewardMATH benchmarks.<n>We propose CRew-DPO, a training strategy that constructs preference data from confidence scores combined with correctness signals.
  arXiv  Detail & Related papers  (2025-10-15T12:51:47Z)
• The Thinking Therapist: Training Large Language Models to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization [0.0]
  Acceptance and Commitment Therapy (ACT) is a cognitive behavioral therapy with emerging evidence of efficacy in several psychiatric conditions.<n>This study investigates the impact of post-training methodology and explicit reasoning on the ability of a small open-weight large language model (LLM) to deliver ACT.
  arXiv  Detail & Related papers  (2025-09-08T02:30:12Z)
• Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects [0.6087817758152709]
  We present a systematic study of personality control using the Big Five traits.<n>Trait-level analysis shows openness as uniquely challenging, agreeableness as most resistant to ICL.<n>Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs.
  arXiv  Detail & Related papers  (2025-09-05T04:19:15Z)
• TriCon-SF: A Triple-Shuffle and Contribution-Aware Serial Federated Learning Framework for Heterogeneous Healthcare Data [22.044689597490123]
  TriCon-SF is a novel serial federated learning framework that integrates triple shuffling and contribution awareness.<n> experiments on non-IID healthcare datasets demonstrate that TriCon-SF outperforms standard serial and parallel federated learning in both accuracy and communication efficiency.
  arXiv  Detail & Related papers  (2025-06-20T03:40:35Z)
• Automated Fidelity Assessment for Strategy Training in Inpatient Rehabilitation using Natural Language Processing [53.096237570992294]
  Strategy training is a rehabilitation approach that teaches skills to reduce disability among those with cognitive impairments following a stroke. Standardized fidelity assessment is used to measure adherence to treatment principles. We developed a rule-based NLP algorithm, a long-short term memory (LSTM) model, and a bidirectional encoder representation from transformers (BERT) model for this task.
  arXiv  Detail & Related papers  (2022-09-14T15:33:30Z)
• Learning Domain Adaptive Object Detection with Probabilistic Teacher [93.76128726257946]
  We present a simple yet effective framework, termed as Probabilistic Teacher (PT) PT aims to capture the uncertainty of unlabeled target data from a gradually evolving teacher and guides the learning of a student in a mutually beneficial manner. We also present a novel Entropy Focal Loss (EFL) to further facilitate the uncertainty-guided self-training.
  arXiv  Detail & Related papers  (2022-06-13T16:24:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.