Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation

• URL: http://arxiv.org/abs/2607.18508v1
• Date: Mon, 20 Jul 2026 21:05:08 GMT
• Title: Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
• Abstract summary: We conduct a systematic shortcut audit of EmoPrefer using content-blind probes.<n>A simple logistic regression using only description length and generator identity performs comparably to LoRA-finetuned 7B text and audio-visual judges.<n>Results do not imply that human preferences are inherently stylistic or that the descriptions contain no emotional information.
• Score: 18.639828982880108
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Preference over model-generated emotion descriptions is emerging as a standard evaluation metric for multimodal emotion understanding, exemplified by the MER2026 MER-Prefer track on EmoPrefer. Such benchmarks assume that predicting the preferred description requires grounded cross-modal understanding of the video. We conduct a systematic shortcut audit of EmoPrefer using content-blind probes. A simple logistic regression using only description length and generator identity, without processing the text, video, or audio, performs comparably to LoRA-finetuned 7B text and audio-visual judges (65.8 versus 66.8 WAF on EmoPrefer-V2). Generator identity is recoverable from description text with 99.5 percent accuracy, every candidate pair contrasts two distinct generators, and the human preference labels agree with a fold-exclusive per-generator win-rate prior on 66 percent of the evaluated pairs. When the human label conflicts with this prior, trained judges still follow the style prior on 63 to 80 percent of the pairs. On a length-matched subset that neutralizes verbosity bias, the tested media configurations yield no statistically significant improvement, while an ODIN-inspired diagnostic that decouples the style shortcut leaves its content head near chance. These results do not imply that human preferences are inherently stylistic or that the descriptions contain no emotional information. Instead, they show that the current scores can be reached without verifying either description against the video. We recommend source-balanced pairing, strict length control, counter-stereotypical sliced reporting, and multi-annotator consensus for future cross-generator evaluations. Code is available at https://github.com/jiabingyang01/EmoPrefer-Audit.

Related papers

• EmoStance: Response-Side Affective-Orientation Control for Empathetic Response Generation via Emoji Weak Supervision [3.916665092534829]
  We use multi-annotator emoji distributions as weak affective--attitudinal evidence, rather than as output symbols or gold labels, to induce a latent control space.<n>We construct EmojiDialogue, an utterance-level extension of EmpatheticDialogues with emoji votes and confidence scores, and propose EmoStance.<n>EmoStance achieves a 62.2% decisive win rate, with the clearest gains in contextual specificity and perceived responsiveness.
  arXiv  Detail & Related papers  (2026-09-02T05:43:14Z)
• Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence [21.413257942637888]
  Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses.<n>We introduce a query-only framework that evolves a rubric set from an empty set without external annotations or model training.<n>Experiments across five benchmark suites demonstrate the effectiveness of preferences on Trial.
  arXiv  Detail & Related papers  (2026-07-16T15:01:33Z)
• Debiasing Without Protected Attributes: Latent Concept Erasure from Textual Profiles [48.07449275130565]
  Most fairness research in NLP assumes direct access to protected attributes such as gender, race, or nationality.<n>This raises a key question: can debiasing succeed without direct access to sensitive attributes?<n>We propose H-SAL, which performs post-hoc concept and attribute erasure using self-description text as an implicit debiasing signal.
  arXiv  Detail & Related papers  (2026-06-10T13:49:27Z)
• Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization [51.93456979139756]
  Free-text explanations extend human label variation beyond label disagreement.<n>We study whether large language models can learn and reproduce such annotator-specific label-explanation behavior.
  arXiv  Detail & Related papers  (2026-05-27T17:55:00Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• RAGR: Review-Augmented Generative Recommendation [61.29879991713178]
  Review-Augmented Generative Recommendation (RAGR) is a novel GR framework that incorporates review feedback directly into the generative user sequence.<n>RAGR yields consistent and significant gains over strong GR backbones across all metrics.
  arXiv  Detail & Related papers  (2026-05-17T05:21:23Z)
• The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking [0.0]
  This paper presents an eight-week observational comparison of 68 single-file HTML generations collected across 17 public experiments in the "HTML AI Battle" project.<n>Four reasoning model families, GPT, Gemini, Grok, and Claude, were compared under a fixed public-interface protocol with no custom instructions, no personality tuning, and no repair prompts.<n> Claude was the strongest and most consistent family, leading mean performance and winning 9/17 prompts under the primary human weighted score.
  arXiv  Detail & Related papers  (2026-05-06T18:19:11Z)
• Relational Preference Encoding in Looped Transformer Internal States [0.0]
  We investigate how looped transformers encode human preference in their internal iteration states.<n>Using Ouro-2.6B-Thinking, a 2.6B- parameter looped transformer with iterative refinement, we extract hidden states from each loop iteration.<n>We train lightweight evaluator heads to predict human preference on the Anthropic HH-RLHF dataset.<n>Our pairwise evaluator achieves 95.2% test accuracy on 8,552 unseen examples, surpassing a full-batch L-BFGS probe (84.5%) while the base model remains completely frozen.
  arXiv  Detail & Related papers  (2026-04-10T20:00:49Z)
• Same Words, Different Judgments: Modality Effects on Preference Alignment [8.352948546053776]
  We compare text and audio evaluations of identical semantic content across 100 prompts.<n>Audio preferences prove as reliable as text, with inter-rater agreement reaching good levels.<n>Audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria.
  arXiv  Detail & Related papers  (2026-02-26T07:34:15Z)
• Towards Empathetic Conversational Recommender Systems [77.53167131692]
  We propose an empathetic conversational recommender (ECR) framework. ECR contains two main modules: emotion-aware item recommendation and emotion-aligned response generation. Our experiments on the ReDial dataset validate the efficacy of our framework in enhancing recommendation accuracy and improving user satisfaction.
  arXiv  Detail & Related papers  (2024-08-30T15:43:07Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)
• Exploring Explainable Selection to Control Abstractive Summarization [51.74889133688111]
  We develop a novel framework that focuses on explainability. A novel pair-wise matrix captures the sentence interactions, centrality, and attribute scores. A sentence-deployed attention mechanism in the abstractor ensures the final summary emphasizes the desired content.
  arXiv  Detail & Related papers  (2020-04-24T14:39:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.