NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs

• URL: http://arxiv.org/abs/2605.29685v1
• Date: Thu, 28 May 2026 09:51:06 GMT
• Title: NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs
• Abstract summary: Large language models (LLMs) are increasingly applied in social contexts such as emotional companionship and customer service.<n>Existing social intelligence benchmarks lack a unified framework that organizes social abilities into a unified structure.<n>NICE is a diagnostic benchmark of 137 items operationalized through representative Chinese contexts.
• Score: 18.51325484272099
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) are increasingly applied in social contexts such as emotional companionship and customer service, measuring their social intelligence has become critical to the quality and safety of human-AI interaction. However, existing social intelligence benchmarks lack a unified framework that organizes social abilities into a unified structure, and therefore cannot enable fine-grained diagnosis. To build the first holistic diagnostic evaluation grounded in social theory, we first construct a social intelligence framework through a literature review and multi-stage expert validation guided by psychometric principles. The resulting framework includes 4 categories and 11 dimensions, each further specified by fine-grained capability facets. Building on this framework, we introduce NICE (Norm, Interaction, Cognition, Experience), a diagnostic benchmark of 137 items operationalized through representative Chinese contexts. Across 5 frontier LLMs and a human reference group, models score higher in aggregate accuracy yet show a consistent weakness in Communication, which the framework localizes to 3 specific capability facets: multi-turn communication, nonverbal communication, and synchrony. NICE thus reframes social intelligence evaluation toward theory-grounded diagnosis of socially consequential weaknesses in LLMs.

Related papers

• SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models [86.19617358080016]
  Social Omni is a benchmark that operationalizes the evaluation of conversational interactivity across three core dimensions.<n>Social Omni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances.<n>Our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions.
  arXiv  Detail & Related papers  (2026-03-17T17:58:44Z)
• Social-R1: Towards Human-like Social Reasoning in LLMs [74.32494331695837]
  We argue that cultivating human-like social intelligence requires training with challenging cases that resist shortcut solutions.<n>We propose Social-R1, a reinforcement learning framework that aligns model reasoning with human cognition through multi-dimensional rewards.
  arXiv  Detail & Related papers  (2026-03-10T06:26:24Z)
• HeartBench: Probing Core Dimensions of Anthropomorphic Intelligence in LLMs [20.794341575633503]
  HeartBench is a framework designed to evaluate the integrated emotional, cultural, and ethical dimensions of Chinese Large Language Models (LLMs)<n>Even leading models achieve only 60% of the expert-defined ideal score.<n>Analysis using a difficulty-stratified Hard Set'' reveals a significant performance decay in scenarios involving subtle emotional subtexts and complex ethical trade-offs.
  arXiv  Detail & Related papers  (2025-12-26T03:54:56Z)
• S$^3$IT: A Benchmark for Spatially Situated Social Intelligence Test [26.79990069295221]
  We introduce the Spatially Situated Social Intelligence Test (S$3$IT), a benchmark specifically designed to evaluate embodied social intelligence.<n>It is centered on a novel and challenging seat-ordering task, requiring an agent to arrange seating in a 3D environment for a group of large language model-driven NPCs.<n>Our framework generates a vast and diverse scenario space with controllable difficulty, compelling the agent to acquire preferences through active dialogue, perceive the environment via autonomous exploration, and perform multi-objective optimization within a complex constraint network.
  arXiv  Detail & Related papers  (2025-12-23T02:36:56Z)
• SocialNLI: A Dialogue-Centric Social Inference Dataset [49.60157928163403]
  We introduce SocialNLI -- the first social dialogue inference dataset.<n>SocialNLI consists of a collection of dialogue transcripts hand-picked to center complex social nuances.<n>We evaluate reasoning models theory-of-mind ability through multi-step counterfactual reasoning.
  arXiv  Detail & Related papers  (2025-10-06T23:42:01Z)
• SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models [41.68365456601248]
  We introduce SocialMaze, a new benchmark specifically designed to evaluate social reasoning.<n>SocialMaze systematically incorporates three core challenges: deep reasoning, dynamic interaction, and information uncertainty.<n>It provides six diverse tasks across three key settings: social reasoning games, daily-life interactions, and digital community platforms.
  arXiv  Detail & Related papers  (2025-05-29T17:47:36Z)
• MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems [17.381122321801556]
  We introduce MetaMind, a multi-agent framework inspired by psychological theories of metacognition.<n>Our framework achieves state-of-the-art performance across three challenging benchmarks, with 35.7% improvement in real-world social scenarios.<n>This work advances AI systems toward human-like social intelligence, with applications in empathetic dialogue and culturally sensitive interactions.
  arXiv  Detail & Related papers  (2025-05-25T02:32:57Z)
• Social Genome: Grounded Social Reasoning Abilities of Multimodal Models [61.88413918026431]
  Social reasoning abilities are crucial for AI systems to interpret and respond to multimodal human communication and interaction within social contexts.<n>We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models.
  arXiv  Detail & Related papers  (2025-02-21T00:05:40Z)
• SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents [73.35393511272791]
  We propose an interactive learning method, SOTOPIA-$pi$, improving the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement training on filtered social interaction data according to large language model (LLM) ratings.
  arXiv  Detail & Related papers  (2024-03-13T17:17:48Z)
• SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents [107.4138224020773]
  We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and humans. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We find that GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills.
  arXiv  Detail & Related papers  (2023-10-18T02:27:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.