Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale

• URL: http://arxiv.org/abs/2508.05938v1
• Date: Fri, 08 Aug 2025 02:04:14 GMT
• Title: Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale
• Abstract summary: Prosociality is a novel and increasingly important challenge for trust and safety systems.<n>Unlike toxic content detection, prosociality lacks well-established definitions and labeled data.<n>We present a practical, three-stage pipeline that enables scalable, high-precision prosocial content classification.
• Score: 42.9822726298117
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Detecting prosociality in text--communication intended to affirm, support, or improve others' behavior--is a novel and increasingly important challenge for trust and safety systems. Unlike toxic content detection, prosociality lacks well-established definitions and labeled data, requiring new approaches to both annotation and deployment. We present a practical, three-stage pipeline that enables scalable, high-precision prosocial content classification while minimizing human labeling effort and inference costs. First, we identify the best LLM-based labeling strategy using a small seed set of human-labeled examples. We then introduce a human-AI refinement loop, where annotators review high-disagreement cases between GPT-4 and humans to iteratively clarify and expand the task definition-a critical step for emerging annotation tasks like prosociality. This process results in improved label quality and definition alignment. Finally, we synthesize 10k high-quality labels using GPT-4 and train a two-stage inference system: a lightweight classifier handles high-confidence predictions, while only $\sim$35\% of ambiguous instances are escalated to GPT-4o. This architecture reduces inference costs by $\sim$70% while achieving high precision ($\sim$0.90). Our pipeline demonstrates how targeted human-AI interaction, careful task formulation, and deployment-aware architecture design can unlock scalable solutions for novel responsible AI tasks.

Related papers

• Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision [2.510555578533785]
  We present a production-scale multi-task ranking system that integrates semantic relevance as a primary optimization objective.<n>Our architecture employs an ordinal relevance head that predicts cumulative probabilities over relevance thresholds.<n>Our approach significantly improves semantic alignment while preserving core engagement objectives.
  arXiv  Detail & Related papers  (2026-05-26T21:26:17Z)
• The Novelty Bottleneck: A Framework for Understanding Human Effort Scaling in AI-Assisted Work [4.726153739634646]
  We propose a model of human-AI collaboration that isolates a mechanism we call the novelty bottleneck.<n>The model assumes that tasks decompose into atomic decisions, a fraction $$ of which are "novel"<n>Our contribution is not a proof that human effort must scale linearly, but a framework that identifies the novelty fraction as the key parameter governing AI-assisted productivity.
  arXiv  Detail & Related papers  (2026-03-28T22:50:13Z)
• SAGE: Scalable AI Governance & Evaluation [10.238041570564395]
  textbfSAGE is a framework that operationalizes high-quality human product judgment as a scalable evaluation signal.<n>SAGE was deployed within LinkedIn Search ecosystems and powered policy oversight that measured ramped model variants and detected regressions invisible to engagement metrics.
  arXiv  Detail & Related papers  (2026-02-08T06:42:50Z)
• Human-Corrected Labels Learning: Enhancing Labels Quality via Human Correction of VLMs Discrepancies [6.58446551781724]
  We propose Human-Corrected Labels (HCLs), a novel setting that efficient human correction for VLM-generated noisy labels.<n>HCLs deploys human correction only for instances with VLM discrepancies, achieving both higher-quality annotations and reduced labor costs.<n>Our approach achieves superior classification performance and is robust to label noise, validating the effectiveness of HCL in practical weak supervision scenarios.
  arXiv  Detail & Related papers  (2025-11-12T07:38:19Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• How Good Are Synthetic Requirements ? Evaluating LLM-Generated Datasets for AI4RE [0.5156484100374059]
  This paper presents an enhanced Product Line approach for generating synthetic requirements data.<n>We investigate four research questions assessing how prompting strategies, automated prompt optimization, and post-generation curation affect data quality.<n>Our results show that synthetic requirements can match or outperform human-authored ones for specific tasks.
  arXiv  Detail & Related papers  (2025-06-26T10:52:07Z)
• Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Values [0.6640968473398455]
  Superego agent steers AI planning by referencing user-selected "Creed Constitutions"<n>System validates plans against these constitutions and a universal ethical floor before execution.
  arXiv  Detail & Related papers  (2025-06-08T20:31:26Z)
• Label-Aware Hyperbolic Embeddings for Fine-grained Emotion Classification [11.973115466838717]
  Fine-grained emotion classification (FEC) is a challenging task. FEC needs to handle subtle nuance between labels, which can be complex and confusing. Most existing models only address text classification problem in the euclidean space. We propose HypEmo, a novel framework that can integrate hyperbolic embeddings to improve the FEC task.
  arXiv  Detail & Related papers  (2023-06-26T16:28:33Z)
• Beyond Active Learning: Leveraging the Full Potential of Human Interaction via Auto-Labeling, Human Correction, and Human Verification [3.58439716487063]
  Active Learning (AL) is a human-in-the-loop framework to interactively and adaptively label data instances. We propose CLARIFIER, an Interactive Learning framework that admits more effective use of human interaction by leveraging the reduced cost of verification.
  arXiv  Detail & Related papers  (2023-06-02T05:40:11Z)
• Active Learning For Contextual Linear Optimization: A Margin-Based Approach [6.09977411428684]
  We introduce a label acquisition algorithm that sequentially decides whether to request the labels'' of feature samples from unlabeled data.<n>We derive upper bounds on the label complexity, defined as the number of samples whose labels are acquired.<n>We show that our algorithm has a much smaller label complexity than the naive supervised learning approach.
  arXiv  Detail & Related papers  (2023-05-11T05:44:36Z)
• Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification [57.36281142038042]
  We present a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts a pretrained neural network on a new task to significantly improve performance. We also present a new training protocol based on Coordinate-Descent called UpperCaSE that exploits meta-trained CaSE blocks and fine-tuning routines for efficient adaptation.
  arXiv  Detail & Related papers  (2022-06-20T15:25:08Z)
• Unsupervised Domain Adaptive Salient Object Detection Through Uncertainty-Aware Pseudo-Label Learning [104.00026716576546]
  We propose to learn saliency from synthetic but clean labels, which naturally has higher pixel-labeling quality without the effort of manual annotations. We show that our proposed method outperforms the existing state-of-the-art deep unsupervised SOD methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-26T16:03:55Z)
• Delving into Probabilistic Uncertainty for Unsupervised Domain Adaptive Person Re-Identification [54.174146346387204]
  We propose an approach named probabilistic uncertainty guided progressive label refinery (P$2$LR) for domain adaptive person re-identification. A quantitative criterion is established to measure the uncertainty of pseudo labels and facilitate the network training. Our method outperforms the baseline by 6.5% mAP on the Duke2Market task, while surpassing the state-of-the-art method by 2.5% mAP on the Market2MSMT task.
  arXiv  Detail & Related papers  (2021-12-28T07:40:12Z)
• Adaptive Self-training for Few-shot Neural Sequence Labeling [55.43109437200101]
  We develop techniques to address the label scarcity challenge for neural sequence labeling models. Self-training serves as an effective mechanism to learn from large amounts of unlabeled data. meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels.
  arXiv  Detail & Related papers  (2020-10-07T22:29:05Z)
• Progressive Identification of True Labels for Partial-Label Learning [112.94467491335611]
  Partial-label learning (PLL) is a typical weakly supervised learning problem, where each training instance is equipped with a set of candidate labels among which only one is the true label. Most existing methods elaborately designed as constrained optimizations that must be solved in specific manners, making their computational complexity a bottleneck for scaling up to big data. This paper proposes a novel framework of classifier with flexibility on the model and optimization algorithm.
  arXiv  Detail & Related papers  (2020-02-19T08:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.