Overview of the PsyDefDetect Shared Task at BioNLP 2026: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations

• URL: http://arxiv.org/abs/2605.24907v1
• Date: Sun, 24 May 2026 07:17:54 GMT
• Title: Overview of the PsyDefDetect Shared Task at BioNLP 2026: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations
• Abstract summary: We present an overview of PsyDefDetect, the shared task on detecting levels of psychological defense mechanisms in emotional support dialogues.<n>The task asks systems to classify a target seeker utterance into one of nine categories: seven hierarchical Defense Mechanism Rating Scales (DMRS) levels plus two auxiliary labels.<n>The best system achieved a macro F1-score of 0.420, surpassing the strongest fine-tuned baseline reported in the dataset paper by a notable margin.
• Score: 33.49851288070166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present an overview of PsyDefDetect, the shared task on detecting levels of psychological defense mechanisms in emotional support dialogues, co-located with BioNLP@ACL 2026. Grounded in the clinically validated Defense Mechanism Rating Scales (DMRS) framework, the task asks systems to classify a target seeker utterance, given its preceding dialogue context, into one of nine categories: seven hierarchical DMRS levels plus two auxiliary labels. Participants worked on PsyDefConv, a newly released corpus of 200 dialogues and 2336 help-seeker utterances annotated under DMRS with substantial inter-annotator agreement. The task attracted 172 participants on CodaBench who produced 563 submissions, with 21 teams officially registering their results for the final ranking. The best system achieved a macro F1-score of 0.420, surpassing the strongest fine-tuned baseline reported in the dataset paper by a notable margin, yet leaving clear headroom. Our analysis highlights (i) a persistent tendency to over-predict the majority High-Adaptive class, (ii) a widening gap between accuracy and macro-F1 that reveals class-imbalance sensitivity, and (iii) the value of theory-aware and LLM-based approaches for fine-grained defensive-function classification. We release all task materials and invite the community to continue work on this novel intersection of clinical psychology and NLP.

Related papers

• From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models [41.36461541603076]
  Large language model (LLM) evaluation spans diverse tasks and benchmarks.<n>Evidence remains organized around tasks rather than the capabilities they probe.<n>We introduce a multi-layer taxonomy of 14 capability domains and 91 subskills.
  arXiv  Detail & Related papers  (2026-07-24T10:49:50Z)
• LinguIUTics at PsyDefDetect: Iterative Imbalance-Aware Fine-tuning of Qwen3-8B for Psychological Defense Mechanism Classification [0.0]
  For the PsyDefDetect 2026 shared task, our team LinguIUTics achieves a macro F1-score of 0.3917 on the official positive-class leaderboard.<n>We leverage three key strategies: grouped stratified cross-validation, minority-class round-robin lexical augmentation, and a post-processing pipeline with logit bias tuning and ensemble blending.
  arXiv  Detail & Related papers  (2026-05-30T09:47:29Z)
• Mitigating Data Scarcity in Psychological Defense Classification with Context-Aware Synthetic Augmentation [3.11997128394121]
  Psychological defense mechanisms (PDMs) are unconscious cognitive processes that modulate how individuals perceive and respond to emotional distress.<n>We propose a context-aware synthetic augmentation framework combined with a hybrid classification model.<n>Our hybrid model integrates contextual language representations with basic clinical features, along with 150 annotated defense items.
  arXiv  Detail & Related papers  (2026-05-14T05:02:34Z)
• UTS at PsyDefDetect: Multi-Agent Councils and Absence-Based Reasoning for Defense Mechanism Classification [1.348563827917271]
  This paper describes our system for classifying psychological defense mechanisms in emotional support dialogues.<n>A central insight is that defense mechanisms are defined by what is absent: missing affect, blocked cognition, denied reality.<n>We encode this as an affect-cognition integration spectrum in prompt-level clinical rules, which account for the largest single gain.
  arXiv  Detail & Related papers  (2026-05-10T21:30:02Z)
• Nürnberg NLP at PsyDefDetect: Multi-Axis Voter Ensembles for Psychological Defence Mechanism Classification [0.0]
  In PsyDefDetect the eight positive defence categories share surface language and differ only in pragmatic function.<n>We translate this insight into a 9-voter ensemble spanning three axes.<n>The system reaches $Ftest=.420$ on the hidden test set, placing first among 21 registered teams.
  arXiv  Detail & Related papers  (2026-05-08T11:32:31Z)
• Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity [66.94391219005291]
  This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic.<n>Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling.<n>ACP achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines.
  arXiv  Detail & Related papers  (2026-03-18T06:15:35Z)
• AILS-NTUA at SemEval-2026 Task 10: Agentic LLMs for Psycholinguistic Marker Extraction and Conspiracy Endorsement Detection [14.20669481486209]
  This paper presents a novel agentic LLM pipeline for SemEval-2026 Task 10 that jointly extracts psycholinguistic conspiracy markers and detects conspiracy endorsement.<n>For marker extraction, we propose Dynamic Discrimi Chain-of-Thought (DD-CoT) with deterministic anchoring to resolve semantic ambiguity and character-level brittleness.<n>For conspiracy detection, an "Anti-Echo Chamber" architecture, consisting of an adversarial Parallel Council adjudicated by a Calibrated Judge, overcomes the "Reporter Trap"
  arXiv  Detail & Related papers  (2026-03-05T08:09:10Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition [56.69875958980474]
  This work considers approaches to defending learned systems and how security defenses result in performance inequities across different sub-populations. We find that many methods that have been proposed can cause direct harm, like false rejection and unequal benefits from robustness training. We present a comparison of equality between two rejection-based defenses: randomized smoothing and neural rejection, finding randomized smoothing more equitable due to the sampling mechanism for minority groups.
  arXiv  Detail & Related papers  (2023-02-17T16:19:26Z)
• RuArg-2022: Argument Mining Evaluation [69.87149207721035]
  This paper is a report of the organizers on the first competition of argumentation analysis systems dealing with Russian language texts. A corpus containing 9,550 sentences (comments on social media posts) on three topics related to the COVID-19 pandemic was prepared. The system that won the first place in both tasks used the NLI (Natural Language Inference) variant of the BERT architecture.
  arXiv  Detail & Related papers  (2022-06-18T17:13:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.