Towards Emotion Consistency Analysis of Large Language Models in Emotional Conversational Contexts

• URL: http://arxiv.org/abs/2605.06476v1
• Date: Thu, 07 May 2026 16:01:48 GMT
• Title: Towards Emotion Consistency Analysis of Large Language Models in Emotional Conversational Contexts
• Abstract summary: We examine the consistency of Large Language Models (LLMs) with respect to their own generated responses in an emotionally-driven conversational context.<n>Our findings indicate that LLMs exhibit below-average performance and remain vulnerable to false beliefs embedded within queries.<n>The results raise important considerations for LLMs' deployment in high-stakes, emotionally sensitive contexts.
• Score: 41.84500687049023
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we conduct an analysis to examine the consistency of Large Language Models (LLMs) with respect to their own generated responses in an emotionally-driven conversational context. Specifically, the text generated by LLM is framed as a query to the same model, and its responses are subsequently assessed. This is performed with three queries across two dimensions of extreme and moderate emotions. The three queries are, in particular, false claim queries that contain inherently wrong assumptions (false presuppositions) in increasing order of intensity. Two commercial models, Claude-3.5-haiku, GPT4o-mini, and a medium-sized model, Mistral-7B, are considered in the study. Our findings indicate that LLMs exhibit below-average performance and remain vulnerable to false beliefs embedded within queries. This susceptibility is especially pronounced for moderate emotional content. Furthermore, an extended attention-score-based analysis highlights a shift in models' priority from evaluative to generative. The results raise important considerations for LLMs' deployment in high-stakes, emotionally sensitive contexts.

Related papers

• MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents [41.829135334587626]
  MERRY is a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents.<n>We transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task.<n>We conduct extensive evaluations based on MERRY.
  arXiv  Detail & Related papers  (2026-02-24T02:53:58Z)
• Evaluating LLM Alignment on Personality Inference from Real-World Interview Data [7.061237517845673]
  Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding.<n>Their ability to interpret human personality traits, a critical aspect of such applications, remains unexplored.<n>We introduce a novel benchmark comprising semi-structured interview transcripts paired with validated continuous Big Five trait scores.
  arXiv  Detail & Related papers  (2025-09-16T16:54:35Z)
• LLMs for Argument Mining: Detection, Extraction, and Relationship Classification of pre-defined Arguments in Online Comments [11.976321200958262]
  We evaluate four state-of-the-art Large Language Models (LLMs) on three argument mining tasks.<n> Quantitative evaluation suggests an overall strong performance across the three tasks.<n>A detailed error analysis revealed systematic shortcomings on long and nuanced comments and emotionally charged language.
  arXiv  Detail & Related papers  (2025-05-29T00:29:51Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models [75.05436691700572]
  We introduce ExpliCa, a new dataset for evaluating Large Language Models (LLMs) in explicit causal reasoning.<n>We tested seven commercial and open-source LLMs on ExpliCa through prompting and perplexity-based metrics.<n>Surprisingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events.
  arXiv  Detail & Related papers  (2025-02-21T14:23:14Z)
• Beyond No: Quantifying AI Over-Refusal and Emotional Attachment Boundaries [0.0]
  We present an open-source benchmark and evaluation framework for assessing emotional boundary handling in Large Language Models (LLMs)<n>We evaluated three leading LLMs on their ability to maintain appropriate emotional boundaries through pattern-matched response analysis.<n>Our framework quantifies responses across seven key patterns: direct refusal, apology, explanation, deflection, acknowledgment, boundary setting, and emotional awareness.
  arXiv  Detail & Related papers  (2025-02-20T19:09:40Z)
• Consistency of Responses and Continuations Generated by Large Language Models on Social Media [11.076748897687764]
  Large Language Models (LLMs) demonstrate remarkable capabilities in text generation, yet their emotional consistency and semantic coherence in social media contexts remain insufficiently understood.<n>This study investigates how LLMs handle emotional content and maintain semantic relationships through continuation and response tasks.
  arXiv  Detail & Related papers  (2025-01-14T13:19:47Z)
• MMRel: A Relation Understanding Benchmark in the MLLM Era [72.95901753186227]
  Multi-Modal Relation Understanding (MMRel) is a benchmark that features large-scale, high-quality, and diverse data on inter-object relations. MMRel is ideal for evaluating MLLMs on relation understanding, as well as for fine-tuning MLLMs to enhance relation comprehension capability.
  arXiv  Detail & Related papers  (2024-06-13T13:51:59Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• What Evidence Do Language Models Find Convincing? [94.90663008214918]
  We build a dataset that pairs controversial queries with a series of real-world evidence documents that contain different facts. We use this dataset to perform sensitivity and counterfactual analyses to explore which text features most affect LLM predictions. Overall, we find that current models rely heavily on the relevance of a website to the query, while largely ignoring stylistic features that humans find important.
  arXiv  Detail & Related papers  (2024-02-19T02:15:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.