Facial-Expression-Aware Prompting for Empathetic LLM Tutoring

• URL: http://arxiv.org/abs/2604.15336v1
• Date: Tue, 10 Mar 2026 08:16:24 GMT
• Title: Facial-Expression-Aware Prompting for Empathetic LLM Tutoring
• Abstract summary: We investigate whether facial-expression-aware signals can improve empathetic tutoring responses through prompt-level integration.<n>We build a scalable simulated tutoring environment where a student agent exhibits diverse facial behaviors from a large unlabeled facial expression video dataset.<n>Across 960 multi-turn conversations spanning three tutor backbones, we evaluate targeted pairwise comparisons with five human raters and an exhaustive AI evaluator.
• Score: 1.4789832988674319
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) enable increasingly capable tutoring-style conversational agents, yet effective tutoring requires sensitivity to learners' affective and cognitive states beyond text alone. Facial expressions provide immediate and practical cues of confusion, frustration, or engagement, but remain underexplored in LLM-driven tutoring. We investigate whether facial-expression-aware signals can improve empathetic tutoring responses through prompt-level integration, without end-to-end retraining. We build a scalable simulated tutoring environment where a student agent exhibits diverse facial behaviors from a large unlabeled facial expression video dataset, and compare four tutor variants: a text-only LLM baseline, a multimodal baseline using a random facial frame, and two Action Unit estimation model (AUM)-based methods that either inject textual AU descriptions or select a peak-expression frame for visual grounding. Across 960 multi-turn conversations spanning three tutor backbones (GPT-5.1, Claude Ops 4.5, and Gemini 2.5 Pro), we evaluate targeted pairwise comparisons with five human raters and an exhaustive AI evaluator. AU-based conditioning consistently improves empathetic responsiveness to facial expressions across all tutor backbones, while AUM-guided peak-frame selection outperforms random-frame visual input. Textual AU abstraction and peak-frame visual injection show model-dependent advantages. Control analyses show that this improvement does not come at the expense of worse pedagogical clarity or responsiveness to textual cues. Finally, AI-human agreement is highest on facial-expression-grounded empathy, supporting scalable AI evaluation for this dimension. Overall, our results show that lightweight, structured facial expression representations can meaningfully enhance empathy in LLM-based tutoring systems with minimal overhead.

Related papers

• Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis [36.79172273160707]
  Facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction.<n>We propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone.<n>We construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations.
  arXiv  Detail & Related papers  (2026-07-27T13:42:37Z)
• LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion [32.661283992568755]
  Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment.<n>Existing approaches often rely on large language models (LLMs) to analyze textual responses of interviewees.<n>We propose an LLM-based framework that semantically fuse facial action units (AUs) with textual responses of AVI.
  arXiv  Detail & Related papers  (2026-06-29T07:35:27Z)
• Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation [63.90351637818903]
  Speech-preserving facial expression manipulation aims to enhance human expressiveness without altering mouth movements tied to the original speech.<n>Current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision.<n>We propose a Personalized Cross-Modal Emotional Correlation Learning algorithm that refines VLM-based supervision.
  arXiv  Detail & Related papers  (2026-04-28T06:02:23Z)
• Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling [2.8037951156321377]
  We propose a multimodal emotion recognition framework for the Expression task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge.<n>Our framework builds on large-scale pre-trained models for visual and audio representation learning and integrates them in a unified multimodal architecture.<n> Experimental results on the ABAW 10th EXPR benchmark demonstrate the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2026-03-12T14:20:29Z)
• Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction [53.12445980832305]
  We propose a facial expression generation method aligned with human preference.<n>A key to our method is framing the generation of identity-independent facial expressions as an action learning process.<n> Experiments on two benchmarks demonstrate that our method effectively aligns facial expressions with human preference.
  arXiv  Detail & Related papers  (2026-03-07T08:03:21Z)
• Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval [15.126709823382539]
  This work advances Contrastive Language-Image Pre-training (CLIP) for person representation learning.<n>We develop a noise-resistant data construction pipeline that leverages the in-context learning capabilities of MLLMs.<n>We introduce the GA-DMS framework, which improves cross-modal alignment by adaptively masking noisy textual tokens.
  arXiv  Detail & Related papers  (2025-09-11T03:06:22Z)
• Multimodal Prompt Alignment for Facial Expression Recognition [24.470095812039286]
  MPA-FER provides fine-grained semantic guidance to the learning process of prompted visual features.<n>Our framework outperforms state-of-the-art methods on three FER benchmark datasets.
  arXiv  Detail & Related papers  (2025-06-26T05:28:57Z)
• VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection [50.57849622045192]
  We propose VAEmo, an efficient framework for emotion-centric joint VA representation learning with external knowledge injection.<n>VAEmo achieves state-of-the-art performance with a compact design, highlighting the benefit of unified cross-modal encoding and emotion-aware semantic guidance.
  arXiv  Detail & Related papers  (2025-05-05T03:00:51Z)
• FaceInsight: A Multimodal Large Language Model for Face Perception [69.06084304620026]
  We propose FaceInsight, a versatile face perception large language model (MLLM) that provides fine-grained facial information.<n>Our approach introduces visual-textual alignment of facial knowledge to model both uncertain dependencies and deterministic relationships among facial information.<n> Comprehensive experiments and analyses across three face perception tasks demonstrate that FaceInsight consistently outperforms nine compared MLLMs.
  arXiv  Detail & Related papers  (2025-04-22T06:31:57Z)
• Compound Expression Recognition via Large Vision-Language Models [9.401699207785015]
  Compound Expression Recognition (CER) is crucial for understanding human emotions and improving human-computer interaction.<n>To address these issues, we propose a novel approach leveraging Large Vision-Language Models (LVLMs)<n>Our method employs a two-stage fine-tuning process: first, pre-trained LVLMs are fine-tuned on basic facial expressions to establish foundational patterns; second, the model is further optimized on a compound-expression dataset to refine visual-language feature interactions.
  arXiv  Detail & Related papers  (2025-03-14T09:46:05Z)
• CIAO! A Contrastive Adaptation Mechanism for Non-Universal Facial Expression Recognition [80.07590100872548]
  We propose Contrastive Inhibitory Adaptati On (CIAO), a mechanism that adapts the last layer of facial encoders to depict specific affective characteristics on different datasets. CIAO presents an improvement in facial expression recognition performance over six different datasets with very unique affective representations.
  arXiv  Detail & Related papers  (2022-08-10T15:46:05Z)
• AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition [79.8779790682205]
  We propose an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-12-29T03:42:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.