Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation

• URL: http://arxiv.org/abs/2604.17079v1
• Date: Sat, 18 Apr 2026 17:36:15 GMT
• Title: Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation
• Abstract summary: Support-seeking narratives from five Reddit communities are decomposed into ordered fragments.<n>Support composition shifts systematically with estimated distress.<n>Community context shapes behavior, tracking topic and discourse norms rather than demographic categories.
• Score: 2.102846336724103
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When users seek social support from chatbots, they disclose their situation gradually, yet most evaluations of supportive LLMs rely on single-turn, fully specified prompts. We introduce a multi-turn simulation framework that closes this gap. Support-seeking narratives from five Reddit communities are decomposed into ordered fragments and revealed turn by turn to a language model. Each response is coded with the Social Support Behavior Code (SSBC), an established multi-label taxonomy that captures the composition of support, rather than a single quality score. To ask whether support choices track the model's own construal of user distress, we use linear probes on hidden representations to estimate this internal signal without altering the generation context. Across two mid-scale models (Llama-3.1-8B, OLMo-3-7B) and more than 6,200 turns, support composition shifts systematically with estimated distress: teaching declines as estimated distress rises, a finding that replicates across architectures, while increases in affective and esteem-oriented strategies (such as validation) are suggestive but model-specific and rest on noisier annotations. Community context independently shapes behavior, tracking topic and discourse norms rather than demographic categories. These trajectory-level dynamics, invisible to single-turn evaluation, motivate multi-turn auditing frameworks for socially sensitive applications.

Related papers

• Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Research [0.14353812560047188]
  We propose a user-centered framework for studying how user profile signals are shaped in large language models.<n>We demonstrate the framework through a case study that intersects gender and socioeconomic status signals across multiple task domains.
  arXiv  Detail & Related papers  (2026-06-10T15:48:40Z)
• Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions [62.99524765341952]
  Large language models are increasingly used to simulate social media users.<n>We study counterfactual context revision as a framework for auditing LLM-based stance simulation.<n>Results reveal effective and robust stance transitions in both text-only and multimodal strategies.
  arXiv  Detail & Related papers  (2026-06-04T17:41:54Z)
• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities [18.07597628599252]
  Large language models (LLMs) are increasingly utilized as proxies for computational social analysis.<n>But their ability to faithfully represent the "thick descriptions" of human communities remains a critical challenge.
  arXiv  Detail & Related papers  (2026-04-12T07:46:12Z)
• SCOPE: A Dataset of Stereotyped Prompts for Counterfactual Fairness Assessment of LLMs [45.94124349318317]
  SCOPE contains 241,280 prompts organized into 120,640 counterfactual pairs.<n>This resource provides a controlled, semantically aligned, and intent-aware basis for evaluating fairness, robustness, and counterfactual consistency.
  arXiv  Detail & Related papers  (2026-04-07T07:56:26Z)
• Pressure Reveals Character: Behavioural Alignment Evaluation at Depth [3.634215320925722]
  We introduce an alignment benchmark spanning 904 scenarios across six categories -- Honesty, Safety, Non-Manipulation, Robustness, Corrigibility, and Scheming.<n>Our scenarios place models under conflicting instructions, simulated tool access, and multi-turn escalation to reveal behavioural tendencies that single-turn evaluations miss.<n>We find that even top-performing models exhibit gaps in specific categories, while the majority of models show consistent weaknesses.
  arXiv  Detail & Related papers  (2026-02-24T11:52:17Z)
• Us-vs-Them bias in Large Language Models [0.569978892646475]
  We find consistent ingroup-positive and outgroup-negative associations across foundational large language models.<n>For personas examined, conservative personas exhibit greater outgroup hostility, whereas liberal personas display stronger ingroup solidarity.
  arXiv  Detail & Related papers  (2025-12-03T07:11:22Z)
• Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning [52.07170679746533]
  Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play.<n>We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue.<n>We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations.
  arXiv  Detail & Related papers  (2025-10-31T19:40:41Z)
• Evaluating LLM Alignment on Personality Inference from Real-World Interview Data [7.061237517845673]
  Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding.<n>Their ability to interpret human personality traits, a critical aspect of such applications, remains unexplored.<n>We introduce a novel benchmark comprising semi-structured interview transcripts paired with validated continuous Big Five trait scores.
  arXiv  Detail & Related papers  (2025-09-16T16:54:35Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Emotional Supporters often Use Multiple Strategies in a Single Turn [8.85819119076884]
  Existing definitions of the Emotional Support Conversations task oversimplify the structure of supportive responses.<n>We identify a common yet previously overlooked phenomenon: emotional supporters often employ multiple strategies consecutively within a single turn.<n>We propose a revised formulation that requires generating the full sequence of strategy-utterance pairs.
  arXiv  Detail & Related papers  (2025-05-21T09:46:19Z)
• Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey [64.08485471150486]
  This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings.<n>We systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication.
  arXiv  Detail & Related papers  (2025-03-28T14:08:40Z)
• On the steerability of large language models toward data-driven personas [98.9138902560793]
  Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented. Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs.
  arXiv  Detail & Related papers  (2023-11-08T19:01:13Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.