Naturalistic measure of social norms alignment

• URL: http://arxiv.org/abs/2605.23420v1
• Date: Fri, 22 May 2026 09:29:18 GMT
• Title: Naturalistic measure of social norms alignment
• Abstract summary: We propose a framework for measuring social norm alignment in naturalistic, free-form settings through solution matching.<n>We construct a dataset of 3k non-trivial social dilemmas in Danish.<n>Our results show that the proposed metrics produce consistent model rankings and reveal variation in agreement across different types of dilemmas.
• Score: 2.3227870457549584
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Social norms reflect shared expectations on acceptable behavior. Measuring social norms alignment remains challenging, with existing approaches typically relying on artificial closed-form evaluations such as multiple-choice questionnaires or measuring agreement with predefined statements. In the context of this work, social norms alignment refers to measuring an agreement between solutions with respect to the social problem or dilemma. We propose a framework for measuring social norm alignment in naturalistic, free-form settings through solution matching. The framework enables us to measure alignment between any two dilemma responses e.g., LLMs to a human, LLMs to LLMs, or human to human. We introduce two metrics: stated and explicit agreement accuracy, and construct a dataset of 3k non-trivial social dilemmas in Danish. All dilemmas are assigned reference solutions derived from three panelists, who serve as culturally grounded judges. We evaluate the agreement of several LLMs and human responses in an interaction setup that resembles natural user-model conversations. Our results show that the proposed metrics produce consistent model rankings and reveal variation in agreement across different types of dilemmas, with higher agreement observed for topics such as neighbor conflicts and shared living situations. Overall, our work introduces a dataset and evaluation framework for studying culturally grounded social reasoning in naturalistic open-ended conversations.

Related papers

• Machine Behavior in Relational Moral Dilemmas: Moral Rightness, Predicted Human Behavior, and Model Decisions [14.29792672073981]
  We characterize machine behavior using the Whistleblower's Dilemma by varying two experimental dimensions: crime severity and relational closeness.<n>Our study evaluates three distinct perspectives: (1) moral rightness (prescriptive norms), (2) predicted human behavior (descriptive social expectations), and (3) autonomous model decision-making.
  arXiv  Detail & Related papers  (2026-04-23T17:14:18Z)
• SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models [86.19617358080016]
  Social Omni is a benchmark that operationalizes the evaluation of conversational interactivity across three core dimensions.<n>Social Omni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances.<n>Our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions.
  arXiv  Detail & Related papers  (2026-03-17T17:58:44Z)
• Social Norm Reasoning in Multimodal Language Models: An Evaluation [0.8181983928344693]
  Multimodal Large Language Models (MLLMs) present promising possibilities to develop software used by robots to identify and reason about norms.<n>This paper investigates the norm reasoning competence of five MLLMs by evaluating their ability to answer norm-related questions based on thirty text-based and thirty image-based stories.<n>Our results show that MLLMs demonstrate superior performance in norm reasoning in text than in images.
  arXiv  Detail & Related papers  (2026-03-03T23:48:21Z)
• MINDS: A Cross-cultural Dialogue Corpus for Social Norm Classification and Adherence Detection [8.551627762137928]
  We present Norm-RAG, a retrieval-augmented, agentic framework for nuanced social norm inference in multi-turn dialogues.<n>Our experiments show that Norm-RAG improves norm detection and generalization, demonstrates improved performance for culturally adaptive and socially intelligent dialogue systems.
  arXiv  Detail & Related papers  (2025-11-13T03:33:39Z)
• EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Norms [0.14323566945483493]
  This study aims to evaluate how opinions are represented in NLP or computational models by introducing an alignment evaluation framework.<n>Our approach models the stance of responses as a proxy for the underlying opinion, enabling a considerate and reflective representation of diverse social viewpoints.
  arXiv  Detail & Related papers  (2025-07-27T13:21:07Z)
• How large language models judge and influence human cooperation [82.07571393247476]
  We assess how state-of-the-art language models judge cooperative actions.<n>We observe a remarkable agreement in evaluating cooperation against good opponents.<n>We show that the differences revealed between models can significantly impact the prevalence of cooperation.
  arXiv  Detail & Related papers  (2025-06-30T09:14:42Z)
• Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks [52.098988739649705]
  This study examines the biases and limitations of LLMs in three roles: answer generator, judge, and debater.<n>We develop a no-consensus'' benchmark by curating examples that encompass a variety of a priori ambivalent scenarios.<n>Our results show that while LLMs can provide nuanced assessments when generating open-ended answers, they tend to take a stance on no-consensus topics when employed as judges or debaters.
  arXiv  Detail & Related papers  (2025-05-28T01:31:54Z)
• Negotiative Alignment: Embracing Disagreement to Achieve Fairer Outcomes -- Insights from Urban Studies [1.3787753513429983]
  Disagreement patterns were systematic in our sample.<n>Group discussion reduced information gaps but not value conflicts.<n>Treating disagreement as signal and reporting worst-group outcomes alongside totals may help planners and AI practitioners surface trade-offs and preserve minority priorities while maintaining efficiency.
  arXiv  Detail & Related papers  (2025-03-16T18:55:54Z)
• EgoNormia: Benchmarking Physical Social Norm Understanding [52.87904722234434]
  EGONORMIA spans seven norm categories: safety, privacy, proxemics, politeness, cooperation, coordination/proactivity, and communication/legibility.<n>Our work demonstrates that current state-of-the-art vision-language models (VLMs) lack robust grounded norm understanding, scoring a maximum of 54% on EGONORMIA and 65% on EGONORMIA-verified.
  arXiv  Detail & Related papers  (2025-02-27T19:54:16Z)
• A Theory of Response Sampling in LLMs: Part Descriptive and Part Prescriptive [53.08398658452411]
  Large Language Models (LLMs) are increasingly utilized in autonomous decision-making.<n>We show that this sampling behavior resembles that of human decision-making.<n>We show that this deviation of a sample from the statistical norm towards a prescriptive component consistently appears in concepts across diverse real-world domains.
  arXiv  Detail & Related papers  (2024-02-16T18:28:43Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Training Socially Aligned Language Models on Simulated Social Interactions [99.39979111807388]
  Social alignment in AI systems aims to ensure that these models behave according to established societal values. Current language models (LMs) are trained to rigidly replicate their training corpus in isolation. This work presents a novel training paradigm that permits LMs to learn from simulated social interactions.
  arXiv  Detail & Related papers  (2023-05-26T14:17:36Z)
• Sociocultural Norm Similarities and Differences via Situational Alignment and Explainable Textual Entailment [31.929550141633218]
  We propose a novel approach to discover and compare social norms across Chinese and American cultures. We build a high-quality dataset of 3,069 social norms aligned with social situations across Chinese and American cultures. To test the ability of models to reason about social norms across cultures, we introduce the task of explainable social norm entailment.
  arXiv  Detail & Related papers  (2023-05-23T19:43:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.