Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities

• URL: http://arxiv.org/abs/2605.27388v1
• Date: Sun, 12 Apr 2026 07:46:12 GMT
• Title: Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities
• Abstract summary: Large language models (LLMs) are increasingly utilized as proxies for computational social analysis.<n>But their ability to faithfully represent the "thick descriptions" of human communities remains a critical challenge.
• Score: 18.07597628599252
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly utilized as proxies for computational social analysis; yet, their ability to faithfully represent the "thick descriptions" (Geertz, 1973) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts. To bridge this gap, we introduce CARE (Community-Aware Reaction Evaluation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news. By characterizing a fine-grained spectrum of illocutionary tones and the underlying attitudes they manifest--validated through human-AI collaboration--our diagnosis reveals a persistent "realism gap": steering LLMs with explicit community prompts fails to inherently improve simulation fidelity. Analysis further identifies divergent behavioral signatures among frontier models, suggesting that current alignment strategies remain insufficient for capturing the sociolinguistic dynamics of online groups.

Related papers

• SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models [86.19617358080016]
  Social Omni is a benchmark that operationalizes the evaluation of conversational interactivity across three core dimensions.<n>Social Omni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances.<n>Our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions.
  arXiv  Detail & Related papers  (2026-03-17T17:58:44Z)
• Interpretable Debiasing of Vision-Language Models for Social Fairness [55.85977929985967]
  We introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in Vision-Language models.<n>We train SAEs on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics.<n>Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.
  arXiv  Detail & Related papers  (2026-02-27T13:37:11Z)
• Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations [2.429376470369691]
  Large Language Models (LLMs) can be successfully leveraged to enhance Agent-Based Modeling.<n>We design a simulator of online social networks enabling a counterfactual simulation where toxic behavior is influenced by moderation interventions.<n>We conduct extensive experiments, unveiling the psychological realism of OSN agents and the superior effectiveness of personalized moderation strategies.
  arXiv  Detail & Related papers  (2025-11-10T15:31:59Z)
• Social Simulations with Large Language Model Risk Utopian Illusion [61.358959720048354]
  We introduce a systematic framework for analyzing large language models' behavior in social simulation.<n>Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions.<n>Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it.
  arXiv  Detail & Related papers  (2025-10-24T06:08:41Z)
• Large Language Model Driven Agents for Simulating Echo Chamber Formation [5.6488384323017]
  The rise of echo chambers on social media platforms has heightened concerns about polarization and the reinforcement of existing beliefs.<n>Traditional approaches for simulating echo chamber formation have often relied on predefined rules and numerical simulations.<n>We present a novel framework that leverages large language models (LLMs) as generative agents to simulate echo chamber dynamics.
  arXiv  Detail & Related papers  (2025-02-25T12:05:11Z)
• Emergence of human-like polarization among large language model agents [79.96817421756668]
  We simulate a networked system involving thousands of large language model agents, discovering their social interactions, result in human-like polarization.<n>Similarities between humans and LLM agents raise concerns about their capacity to amplify societal polarization, but also hold the potential to serve as a valuable testbed for identifying plausible strategies to mitigate polarization and its consequences.
  arXiv  Detail & Related papers  (2025-01-09T11:45:05Z)
• Decoding Echo Chambers: LLM-Powered Simulations Revealing Polarization in Social Networks [12.812531689189065]
  Impact of social media on critical issues such as echo chambers needs to be addressed.<n>Traditional research often oversimplifies emotional tendencies and opinion evolution into numbers and formulas.<n>We propose an LLM-based simulation for the social opinion network to evaluate and counter polarization phenomena.
  arXiv  Detail & Related papers  (2024-09-28T12:49:02Z)
• Social Debiasing for Fair Multi-modal LLMs [59.61512883471714]
  Multi-modal Large Language Models (MLLMs) have dramatically advanced the research field and delivered powerful vision-language understanding capabilities.<n>These models often inherit deep-rooted social biases from their training data, leading to uncomfortable responses with respect to attributes such as race and gender.<n>This paper addresses the issue of social biases in MLLMs by introducing a comprehensive counterfactual dataset with multiple social concepts.
  arXiv  Detail & Related papers  (2024-08-13T02:08:32Z)
• Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents [18.961470450132637]
  This paper emphasizes the importance of spontaneous phenomena, wherein agents deeply engage in contexts and make adaptive decisions without explicit directions. We explored spontaneous cooperation across three competitive scenarios and successfully simulated the gradual emergence of cooperation.
  arXiv  Detail & Related papers  (2024-02-19T18:00:53Z)
• Training Socially Aligned Language Models on Simulated Social Interactions [99.39979111807388]
  Social alignment in AI systems aims to ensure that these models behave according to established societal values. Current language models (LMs) are trained to rigidly replicate their training corpus in isolation. This work presents a novel training paradigm that permits LMs to learn from simulated social interactions.
  arXiv  Detail & Related papers  (2023-05-26T14:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.