TAG: Thinking with Action Unit Grounding for Facial Expression Recognition

• URL: http://arxiv.org/abs/2602.18763v1
• Date: Sat, 21 Feb 2026 09:00:52 GMT
• Title: TAG: Thinking with Action Unit Grounding for Facial Expression Recognition
• Abstract summary: We propose TAG (Thinking with Action Unit Grounding), a vision--language framework that explicitly constrains multimodal reasoning to be supported by facial Action Units (AUs)<n> evaluated on RAF-DB, FERPlus, and AffectNet.
• Score: 18.35567824514563
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Facial Expression Recognition (FER) is a fine-grained visual understanding task where reliable predictions require reasoning over localized and meaningful facial cues. Recent vision--language models (VLMs) enable natural language explanations for FER, but their reasoning is often ungrounded, producing fluent yet unverifiable rationales that are weakly tied to visual evidence and prone to hallucination, leading to poor robustness across different datasets. We propose TAG (Thinking with Action Unit Grounding), a vision--language framework that explicitly constrains multimodal reasoning to be supported by facial Action Units (AUs). TAG requires intermediate reasoning steps to be grounded in AU-related facial regions, yielding predictions accompanied by verifiable visual evidence. The model is trained via supervised fine-tuning on AU-grounded reasoning traces followed by reinforcement learning with an AU-aware reward that aligns predicted regions with external AU detectors. Evaluated on RAF-DB, FERPlus, and AffectNet, TAG consistently outperforms strong open-source and closed-source VLM baselines while simultaneously improving visual faithfulness. Ablation and preference studies further show that AU-grounded rewards stabilize reasoning and mitigate hallucination, demonstrating the importance of structured grounded intermediate representations for trustworthy multimodal reasoning in FER. The code will be available at https://github.com/would1920/FER_TAG .

Related papers

• AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes [59.134503227559634]
  Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding.<n>Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language.<n>We propose Affect Omni, a framework for verifiable affective reasoning.
  arXiv  Detail & Related papers  (2026-08-24T11:25:04Z)
• Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning [12.622982203412478]
  Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs)<n>We propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks.<n>A VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness.<n> Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.
  arXiv  Detail & Related papers  (2026-06-29T08:58:33Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition [51.70817823155725]
  We propose a novel landmark-guided contrastive learning network with vision-language enhancement for Facial Expression Recognition (FER)<n>LaCoVL-FER integrates geometric priors from facial landmarks and semantic priors from a vision-language model.<n>Experiments demonstrate that LaCoVL-FER outperforms state-of-the-art methods on three representative real-world FER datasets.
  arXiv  Detail & Related papers  (2026-05-19T13:15:41Z)
• ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both [55.182037225013836]
  ATLAS is a framework in which a single discrete 'word', termed as a functional token, serves both as an agentic operation and a latent visual reasoning unit.<n> ATLAS achieves superior performance on challenging benchmarks while maintaining clear interpretability.
  arXiv  Detail & Related papers  (2026-05-14T17:59:55Z)
• ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning [55.562958315741646]
  We propose ActFER, an agentic framework that reformulates FER as active visual evidence acquisition followed by multimodal reasoning.<n>We further develop Utility-Calibrated GRPO, a reinforcement learning algorithm tailored to agentic FER.
  arXiv  Detail & Related papers  (2026-04-10T05:53:19Z)
• Hierarchical Vision-Language Interaction for Facial Action Unit Detection [44.02409932746335]
  We propose a Hierarchical Vision-language Interaction for AU Understanding (HiVA) method to enhance AU detection.<n>HiVA employs a large language model to generate diverse and contextually rich AU descriptions to strengthen language-based representation learning.<n> experiments show that HiVA consistently surpasses state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-02-16T03:22:05Z)
• Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy [75.66913260900726]
  Reinforcement Learning with Verifiable Rewards has significantly advanced reasoning capabilities in Large Language Models.<n>Existing paradigms, driven by text-centric outcome rewards, encourage models to bypass visual perception.<n>We propose textbfThinking with Deltas, a framework driven by a textbfDifferential Visual Reasoning Policy.
  arXiv  Detail & Related papers  (2026-01-11T08:25:34Z)
• Stable Language Guidance for Vision-Language-Action Models [62.80963701282789]
  Residual Semantic Steering is a probabilistic framework that disentangles physical affordance from semantic execution.<n> RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.
  arXiv  Detail & Related papers  (2026-01-07T16:16:10Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• Look As You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning [55.232400251303794]
  Look As You Think (LAT) is a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution.<n>LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in IoU@0.5.
  arXiv  Detail & Related papers  (2025-11-15T02:50:23Z)
• Rethinking Occlusion in FER: A Semantic-Aware Perspective and Go Beyond [10.015531203047598]
  We present ORSANet, which introduces auxiliary multi-modal semantic guidance to disambiguate facial occlusion.<n>We also introduce facial landmarks as sparse geometric prior to mitigate intrinsic noises in FER, such as identity and gender biases.<n>Our proposed ORSANet achieves SOTA recognition performance.
  arXiv  Detail & Related papers  (2025-07-21T09:04:29Z)
• Anticipating the Unseen Discrepancy for Vision and Language Navigation [63.399180481818405]
  Vision-Language Navigation requires the agent to follow natural language instructions to reach a specific target. The large discrepancy between seen and unseen environments makes it challenging for the agent to generalize well. We propose Unseen Discrepancy Anticipating Vision and Language Navigation (DAVIS) that learns to generalize to unseen environments via encouraging test-time visual consistency.
  arXiv  Detail & Related papers  (2022-09-10T19:04:40Z)
• AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition [79.8779790682205]
  We propose an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-12-29T03:42:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.