PSK@EEUCA 2026: Fine-Tuning Large Language Models with Synthetic Data Augmentation for Multi-Class Toxicity Detection in Gaming Chat

• URL: http://arxiv.org/abs/2605.07201v1
• Date: Fri, 08 May 2026 03:48:31 GMT
• Title: PSK@EEUCA 2026: Fine-Tuning Large Language Models with Synthetic Data Augmentation for Multi-Class Toxicity Detection in Gaming Chat
• Abstract summary: This paper describes our system for the EEUCA 2026 Shared Task on Understanding Toxic Behavior in Gaming Communities.<n>The task involves classifying World of Tanks chat messages into six toxicity categories: Non-toxic, Insults/Flaming, Other Offensive, Hate/Harassment, Threats, and Extremism.<n>We explore multiple approaches including encoder-based models, instruction-tuned LLMs with LoRA fine-tuning, hierarchical classification, one-vs-rest strategies, and various ensemble methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper describes our system for the EEUCA 2026 Shared Task on Understanding Toxic Behavior in Gaming Communities. The task involves classifying World of Tanks chat messages into six toxicity categories: Non-toxic, Insults/Flaming, Other Offensive, Hate/Harassment, Threats, and Extremism. We explore multiple approaches including encoder-based models, instruction-tuned LLMs with LoRA fine-tuning, hierarchical classification, one-vs-rest strategies, and various ensemble methods. Our best system combines Llama 3.1 8B with carefully calibrated 5\% synthetic data augmentation, achieving an F1-macro score of 0.6234 on the test set, placing 4th out of 35 participating teams. We provide extensive analysis of the dataset's annotation patterns and their impact on model generalization, revealing a critical ''validation trap'' phenomenon where high validation performance correlates with poor test transfer.

Related papers

• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• ShriNep@EEUCA 2026: RAKSHAK - Multi-Task DeBERTa with Rationale Distillation and Jigsaw-Augmented Training for Toxic Intent Classification [0.0]
  This paper presents two systems for classifying World of Tanks chat utterances into six fine-grained toxic intent categories.<n>RakSHAK is a multi-task framework combining rationale distillation from Qwen2.5-14B, Supervised Contrastive Loss, and dedicated rare-class binary heads.<n>M1 fine-tunes DeBERTa-v3-base with Focal Loss on the original GameTox data plus the same 100 extremism samples, without Jigsaw transfer.
  arXiv  Detail & Related papers  (2026-05-14T10:40:27Z)
• thaulab@EEUCA 2026: Who Said What to Whom? A Targeting-Aware Neural-Symbolic Pipeline for Gaming Toxicity Detection [4.187230531998879]
  This paper describes our system for the EEUCA 2026 Shared Task on toxicity classification in gaming chat.<n>Our system achieves a Macro F1 of 0.6441 and accuracy of 0.9062 on the official test set, ranking 3rd in Macro F1 and 1st in accuracy among all teams.
  arXiv  Detail & Related papers  (2026-05-13T15:15:52Z)
• ToxiShield: Promoting Inclusive Developer Communication through Real-Time Toxicity Filtering [1.9500886760872191]
  ToxiShield is a browser extension for GitHub pull requests built using three modules.<n>Toxicity Filter -- to identify whether a text is toxic, Communication coach -- to facilitate just-in-time fine-grained toxicity categorization, and The Reframer -- that generates a revised, constructive alternative.<n>We trained and evaluated multiple deep learning and Large Language Models to identify the best choice for each module.
  arXiv  Detail & Related papers  (2026-04-15T20:46:40Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• <think> So let's replace this phrase with insult... </think> Lessons learned from generation of toxic texts with LLMs [60.169913160819]
  This paper explores the possibility of using synthetic toxic data as an alternative to human-generated data for training models for detoxification.<n>Experiments show that models fine-tuned on synthetic data consistently perform worse than those trained on human data.<n>The root cause is identified as a critical lexical diversity gap: LLMs generate toxic content using a small, repetitive vocabulary of insults that fails to capture the nuances and variety of human toxicity.
  arXiv  Detail & Related papers  (2025-09-10T07:48:24Z)
• CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom [45.382739152668954]
  Distilling advanced Large Language Models' instruction-following capabilities into smaller models has become a mainstream approach in model training.<n>We investigate more diverse signals to capture comprehensive instruction-response pair characteristics.<n>We propose CrowdSelect, an integrated metric incorporating a clustering-based approach to maintain response diversity.
  arXiv  Detail & Related papers  (2025-03-03T18:56:44Z)
• STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions [6.19084217044276]
  Mitigating explicit and implicit biases in Large Language Models (LLMs) has become a critical focus in the field of natural language processing.<n>We introduce the Sensitivity Testing on Offensive Progressions dataset, which includes 450 offensive progressions containing 2,700 unique sentences.<n>Our findings reveal that even the best-performing models detect bias inconsistently, with success rates ranging from 19.3% to 69.8%.
  arXiv  Detail & Related papers  (2024-09-20T18:34:38Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)
• UTNLP at SemEval-2021 Task 5: A Comparative Analysis of Toxic Span Detection using Attention-based, Named Entity Recognition, and Ensemble Models [6.562256987706127]
  This paper presents our team's, UTNLP, methodology and results in the SemEval-2021 shared task 5 on toxic spans detection. The experiments start with keyword-based models and are followed by attention-based, named entity-based, transformers-based, and ensemble models. Our best approach, an ensemble model, achieves an F1 of 0.684 in the competition's evaluation phase.
  arXiv  Detail & Related papers  (2021-04-10T13:56:03Z)
• Lone Pine at SemEval-2021 Task 5: Fine-Grained Detection of Hate Speech Using BERToxic [2.4815579733050153]
  This paper describes our approach to the Toxic Spans Detection problem. We propose BERToxic, a system that fine-tunes a pre-trained BERT model to locate toxic text spans in a given text. Our system significantly outperformed the provided baseline and achieved an F1-score of 0.683, placing Lone Pine in the 17th place out of 91 teams in the competition.
  arXiv  Detail & Related papers  (2021-04-08T04:46:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.