BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts

• URL: http://arxiv.org/abs/2606.10061v1
• Date: Mon, 08 Jun 2026 18:37:23 GMT
• Title: BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
• Abstract summary: Existing sycophancy research primarily focuses on factual agreement and instruction-following settings.<n>We introduce BenSyc, the first benchmark for studying conversational sycophancy in Bengali social contexts.
• Score: 18.592702368344344
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) increasingly participate in emotionally sensitive social conversations, where responses may shift from balanced support toward excessive validation or escalatory alignment. Existing sycophancy research primarily focuses on factual agreement and instruction-following settings, leaving culturally grounded conversational sycophancy underexplored. We introduce BenSyc, the first benchmark for studying conversational sycophancy in Bengali social contexts. Starting from 11,840 Reddit posts and 170k comments collected from communities across Bangladesh and West Bengal, we construct a human-validated benchmark with binary labels and a fine-grained five-level taxonomy spanning Invalidation, Neutral, Support, Validation, and Escalation. We evaluate more than 15 open and proprietary LLMs on conversational alignment classification and response generation tasks. Results show that distinguishing empathetic support from reinforcement-oriented validation remains challenging even for frontier instruction-tuned models: the best system achieves only 61.8 Macro-F1 on binary detection and 61.7 Macro-F1 on five-class classification. In generation settings, several models frequently produce strongly validating or escalatory responses in emotionally charged situations. Our findings highlight substantial variation across model families and conversational behaviors, underscoring the importance of culturally grounded multilingual benchmarks for evaluating socially aligned conversational AI systems.

Related papers

• CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia [72.20656906184539]
  We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue.<n>The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues.<n>Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment.
  arXiv  Detail & Related papers  (2026-08-28T14:56:46Z)
• AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes [59.134503227559634]
  Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding.<n>Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language.<n>We propose Affect Omni, a framework for verifiable affective reasoning.
  arXiv  Detail & Related papers  (2026-08-24T11:25:04Z)
• Learning in Blocks: A Multi Agent Debate Assisted Personalized Adaptive Learning Framework for Language Learning [0.0]
  We introduce Learning in Blocks, a framework that grounds progression in demonstrated conversational competence evaluated using CEFR-aligned rubrics.<n>We benchmark four scoring and recommendation methods on CEFR A2 conversations annotated by ESL experts.<n>An 8-week study with 180 CEFR A2 learners demonstrates that combining rubric-aligned scoring and recommendation with spaced review and mastery-based progression produces better learning outcomes than feedback alone.
  arXiv  Detail & Related papers  (2026-03-29T09:38:41Z)
• BANGLASOCIALBENCH: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction [0.0]
  We introduce BANGLASOCIALBENCH, the first benchmark designed to evaluate sociopragmatic competence in Bangla through context-dependent language use.<n>The benchmark spans three domains: Bangla Address Terms, Kinship Reasoning, and Social Customs, and consists of 1,719 culturally grounded instances written and verified by native Bangla speakers.<n>Our findings show that sociopragmatic failures are often structured and non-random, revealing persistent limitations in how current LLMs infer and apply culturally appropriate language use.
  arXiv  Detail & Related papers  (2026-03-16T21:58:32Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture [5.215285027585101]
  Bengali is spoken by over 285 million people and ranked 6th globally.<n>Existing ethics benchmarks are largely English-centric and shaped by Western frameworks.<n>We introduce BengaliMoralBench, the first large-scale ethics benchmark for the Bengali language and socio-cultural contexts.
  arXiv  Detail & Related papers  (2025-11-05T04:55:35Z)
• From Polyester Girlfriends to Blind Mice: Creating the First Pragmatics Understanding Benchmarks for Slovene [0.12277343096128711]
  We introduce SloPragEval and SloPragMega, the first pragmatics understanding benchmarks for Slovene.<n>We discuss the difficulties of translation, describe the campaign to establish a human baseline, and report pilot evaluations with LLMs.<n>Our results indicate that current models have greatly improved in understanding nuanced language but may still fail to infer implied speaker meaning in non-literal utterances.
  arXiv  Detail & Related papers  (2025-10-24T15:43:42Z)
• ALIGN: Word Association Learning for Cross-Cultural Generalization in Large Language Models [0.8999666725996975]
  It remains a challenge to model and align culture due to limited cultural knowledge.<n>We introduce parameter-efficient fine-tuning on native speakers' free word-association norms.<n>Our work shows that a few million culture-grounded associations can instill value alignment without costly retraining.
  arXiv  Detail & Related papers  (2025-08-19T00:55:20Z)
• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models [121.03333569013148]
  We introduce VisuLogic: a benchmark of 1,000 human-verified problems across six categories.<n>These types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives.<n>Most models score below 30% accuracy-only slightly above the 25% random baseline and far below the 51.4% achieved by humans.
  arXiv  Detail & Related papers  (2025-04-21T17:59:53Z)
• Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Models [45.63440666848143]
  Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities.<n>Despite their success, MLLMs remain vulnerable to conversational adversarial inputs.<n>We study gaslighting negation attacks: a phenomenon where models, despite initially providing correct answers, are persuaded by user-provided negations to reverse their outputs.
  arXiv  Detail & Related papers  (2025-01-31T10:37:48Z)
• SADAS: A Dialogue Assistant System Towards Remediating Norm Violations in Bilingual Socio-Cultural Conversations [56.31816995795216]
  Socially-Aware Dialogue Assistant System (SADAS) is designed to ensure that conversations unfold with respect and understanding. Our system's novel architecture includes: (1) identifying the categories of norms present in the dialogue, (2) detecting potential norm violations, (3) evaluating the severity of these violations, and (4) implementing targeted remedies to rectify the breaches.
  arXiv  Detail & Related papers  (2024-01-29T08:54:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.