CAIA in Practice: Field Evaluation of an AI-Assisted Support System for Text-Based Online Counselling

• URL: http://arxiv.org/abs/2608.22251v1
• Date: Sun, 23 Aug 2026 07:22:05 GMT
• Title: CAIA in Practice: Field Evaluation of an AI-Assisted Support System for Text-Based Online Counselling
• Abstract summary: CAIA is a co-designed AI-based tool suite that demonstrates responsible AI integration into counselling practice.<n>A field evaluation involved 34 professional counsellors conducting authentic sessions with trained student counsellees.
• Score: 0.27988103729716907
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rising global demand for mental health support creates significant service delivery challenges, with asynchronous email counselling serving as a crucial low-threshold channel for accessing care. This paper presents CAIA, a co-designed AI-based tool suite that demonstrates responsible AI integration into counselling practice through seven LLM-driven functions enhanced by retrieval-augmented generation. A field evaluation involved 34 professional counsellors conducting authentic sessions with trained student counsellees (36 threads, 321 messages, 1,257 AI outputs). User behaviour analysis confirms substantial adoption, revealing that professional autonomy and information accuracy are decisive for sustained acceptance, with counsellors particularly valuing interpretive functionalities that provide new perspectives and stimulate professional reflection.

Related papers

• Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs [28.919083157390464]
  We present a comprehensive pipeline for the multi-dimensional evaluation of human counselors' interventions targeting client resistance in text-based therapy.<n>We introduce a theory-driven framework that decomposes counselor responses into four distinct communication mechanisms.<n>We show that our approach can effectively distinguish the quality of different communication mechanisms.
  arXiv  Detail & Related papers  (2026-02-25T07:05:05Z)
• CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models [19.027335814014528]
  We present CARE, an LLM-based framework to automatically predict multi-dimensional alliance scores and generate interpretable rationales from counseling transcripts.<n> CARE is built on the CounselingWAI dataset and enriched with 9,516 expert-curated rationales.<n>Experiments show that CARE outperforms leading LLMs and substantially reduces the gap between counselor evaluations and client-perceived alliance.
  arXiv  Detail & Related papers  (2026-02-24T07:52:56Z)
• Responsible Evaluation of AI for Mental Health [72.85175110624736]
  Current approaches to evaluating AI tools in mental health care are fragmented and poorly aligned with clinical practice, social context, and first-hand user experience.<n>This paper argues for a rethinking of responsible evaluation by introducing an interdisciplinary framework that integrates clinical soundness, social context, and equity.
  arXiv  Detail & Related papers  (2026-01-20T12:55:10Z)
• PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Support [18.251267901872886]
  Large language models (LLMs) are increasingly used for mental health support.<n>LLMs can produce responses that are overly directive, inconsistent, or clinically misaligned.<n>We introduce PAIR-SAFE, a paired-agent framework for auditing and refining AI-generated mental health support.
  arXiv  Detail & Related papers  (2026-01-19T06:20:57Z)
• EmoStage: A Framework for Accurate Empathetic Response Generation via Perspective-Taking and Phase Recognition [1.4436965372953483]
  EmoStage is a framework that enhances empathetic response generation.<n>Our framework introduces perspective-taking to infer clients' psychological states and support needs.<n>Phase recognition is incorporated to ensure alignment with the counseling process.
  arXiv  Detail & Related papers  (2025-06-24T03:18:37Z)
• LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation [68.26911563519162]
  Legal consultation is essential for safeguarding individual rights and ensuring access to justice.<n>Current systems fall short in handling the interactive and knowledge-intensive nature of real-world consultations.<n>LeCoDe is a real-world multi-turn benchmark dataset comprising 3,696 legal consultation dialogues with 110,008 dialogue turns.
  arXiv  Detail & Related papers  (2025-05-26T08:24:32Z)
• Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback [51.26493826461026]
  We propose Psi-Arena, an interactive framework for comprehensive assessment and optimization of large language models (LLMs)<n>Arena features realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients.<n>Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives.
  arXiv  Detail & Related papers  (2025-05-06T08:22:51Z)
• CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy [67.23830698947637]
  We propose a new benchmark, CBT-BENCH, for the systematic evaluation of cognitive behavioral therapy (CBT) assistance.<n>We include three levels of tasks in CBT-BENCH: I: Basic CBT knowledge acquisition, with the task of multiple-choice questions; II: Cognitive model understanding, with the tasks of cognitive distortion classification, primary core belief classification, and fine-grained core belief classification; III: Therapeutic response generation, with the task of generating responses to patient speech in CBT therapy sessions.<n> Experimental results indicate that while LLMs perform well in reciting CBT knowledge, they fall short in complex real-world scenarios
  arXiv  Detail & Related papers  (2024-10-17T04:52:57Z)
• Bridging the Skills Gap: Evaluating an AI-Assisted Provider Platform to Support Care Providers with Empathetic Delivery of Protocolized Therapy [5.6317091933490495]
  Despite high prevalence and burden of mental health conditions, there is a global shortage of mental health providers. We developed AI-Assisted Provider Platform (A2P2), a text-based virtual therapy interface that includes a response feature. We studied providers with and without expertise in mental health treatment delivering a therapy session using the platform with (intervention) and without (control) AI-assistance features.
  arXiv  Detail & Related papers  (2024-01-08T02:23:17Z)
• K-ESConv: Knowledge Injection for Emotional Support Dialogue Systems via Prompt Learning [83.19215082550163]
  We propose K-ESConv, a novel prompt learning based knowledge injection method for emotional support dialogue system. We evaluate our model on an emotional support dataset ESConv, where the model retrieves and incorporates knowledge from external professional emotional Q&A forum.
  arXiv  Detail & Related papers  (2023-12-16T08:10:10Z)
• Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback [46.70617195649979]
  CARE is an AI-based tool to empower and train peer counselors through practice and feedback.<n> CARE helps diagnose which counseling strategies are needed in a given situation and suggests example responses to counselors during their practice sessions.
  arXiv  Detail & Related papers  (2023-05-15T19:48:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.