LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations

• URL: http://arxiv.org/abs/2606.14600v1
• Date: Fri, 12 Jun 2026 16:23:00 GMT
• Title: LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations
• Abstract summary: We introduce LoSoNA, a benchmark for local social norm adaptation in multi-party chat.<n>Each scenario gives a subject model a curated group-chat transcript in which non-subject participants demonstrate a hidden local norm.<n>We evaluate eight frontier and open-weight models under four prompting conditions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online group chats are social spaces with local conversational norms that are rarely stated explicitly. The ability and willingness of LLM-based agents to recognize and adapt to these norms remains mostly unexplored. We introduce LoSoNA, a benchmark for local social norm adaptation in multi-party chat. Each scenario gives a subject model a curated group-chat transcript in which non-subject participants demonstrate a hidden local norm, followed by a final elicitor turn that forces a response revealing whether the subject has inferred that norm. We evaluate eight frontier and open-weight models under four prompting conditions that vary how explicitly the model is told to treat the prior conversation as evidence for how it should answer. Naive prompting remains limited for most models; explicit norm-aware prompting helps unevenly, with Gemini 3.1 Pro reaching $84.2\%$ and Claude Fable 5 reaching $81.6\%$, while several other models show small gains or regressions. LoSoNA contributes to recent calls for evaluating LLM social capabilities by testing whether models can infer local conversational norms from precedent and use them in a one-turn group-chat response.

Related papers

• xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems [57.62172039072062]
  We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities.<n>The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements.<n>The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points.
  arXiv  Detail & Related papers  (2026-09-07T17:30:58Z)
• SCENE: Recognizing Social Norms and Sanctioning in Group Chats [0.0]
  We introduce SCENE, a social-interaction benchmark focused on implicit norms and social sanctioning in multi-party chat.<n>SCENE generates plausible non-roleplay scenarios with scripted personas that follow a hidden norm.<n>We propose behavioral evaluation metrics for two functional adaptation abilities: responsiveness to negative sanctioning, and adapting norm from peers behavior.
  arXiv  Detail & Related papers  (2026-05-08T14:53:04Z)
• Social Norm Reasoning in Multimodal Language Models: An Evaluation [0.8181983928344693]
  Multimodal Large Language Models (MLLMs) present promising possibilities to develop software used by robots to identify and reason about norms.<n>This paper investigates the norm reasoning competence of five MLLMs by evaluating their ability to answer norm-related questions based on thirty text-based and thirty image-based stories.<n>Our results show that MLLMs demonstrate superior performance in norm reasoning in text than in images.
  arXiv  Detail & Related papers  (2026-03-03T23:48:21Z)
• Where Norms and References Collide: Evaluating LLMs on Normative Reasoning [3.8431932182760296]
  Embodied agents, such as robots, will need to interact in situated environments where successful communication often depends on reasoning over social norms.<n>It remains unclear whether Large Language Models (LLMs) can support this kind of reasoning.<n>We introduce SNIC (Situated Norms in Context), a human-validated diagnostic testbed designed to probe how well state-of-the-art LLMs can extract and utilize normative principles relevant to NBRR.
  arXiv  Detail & Related papers  (2026-02-03T01:23:22Z)
• The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior [32.02851847409678]
  We show that identical benchmark questions to the same language model can produce markedly different responses when prompted to a state-less system.<n>We compare offline evaluations to field evaluations conducted by having 800 real users of ChatGPT and Gemini pose benchmark and other provided questions to their chat interfaces.
  arXiv  Detail & Related papers  (2025-09-18T20:41:20Z)
• EgoNormia: Benchmarking Physical Social Norm Understanding [52.87904722234434]
  EGONORMIA spans seven norm categories: safety, privacy, proxemics, politeness, cooperation, coordination/proactivity, and communication/legibility.<n>Our work demonstrates that current state-of-the-art vision-language models (VLMs) lack robust grounded norm understanding, scoring a maximum of 54% on EGONORMIA and 65% on EGONORMIA-verified.
  arXiv  Detail & Related papers  (2025-02-27T19:54:16Z)
• The Art of Saying No: Contextual Noncompliance in Language Models [123.383993700586]
  We introduce a comprehensive taxonomy of contextual noncompliance describing when and how models should not comply with user requests. Our taxonomy spans a wide range of categories including incomplete, unsupported, indeterminate, and humanizing requests. To test noncompliance capabilities of language models, we use this taxonomy to develop a new evaluation suite of 1000 noncompliance prompts.
  arXiv  Detail & Related papers  (2024-07-02T07:12:51Z)
• Self-Exploring Language Models: Active Preference Elicitation for Online Alignment [88.56809269990625]
  We propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, Self-Exploring Language Models (SELM) significantly boosts the performance on instruction-following benchmarks.
  arXiv  Detail & Related papers  (2024-05-29T17:59:07Z)
• NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models [26.64843536942309]
  Large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them.<n>We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability.<n>We create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries.
  arXiv  Detail & Related papers  (2024-04-18T18:48:50Z)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [76.21004582932268]
  We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Arena, a crowdsourced battle platform.
  arXiv  Detail & Related papers  (2023-06-09T05:55:52Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)
• NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly [61.77957329364812]
  We introduce a framework for addressing the novel task of conversation-grounded multi-lingual, multi-cultural norm discovery. NormSAGE elicits knowledge about norms through directed questions representing the norm discovery task and conversation context. It further addresses the risk of language model hallucination with a self-verification mechanism ensuring that the norms discovered are correct.
  arXiv  Detail & Related papers  (2022-10-16T18:30:05Z)
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics [66.96150429230035]
  We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models.
  arXiv  Detail & Related papers  (2021-02-02T18:42:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.