Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

• URL: http://arxiv.org/abs/2607.10428v1
• Date: Sat, 11 Jul 2026 18:22:49 GMT
• Title: Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
• Abstract summary: EYT-Bench is a human-centered benchmark for evaluating large language models.<n> Personas are sampled from public human-curated corpora.<n>State-of-the-art closed and open-source models are statistically close on subjective dimensions.
• Score: 15.994164469249727
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating large language models (LLMs) as multi-turn conversational partners requires probing capabilities that single-turn benchmarks miss: persona consistency, evolving intent tracking, emotional dynamics, and goal completion. We introduce EYT-Bench, a human-centered benchmark built around a three-party decoupled design: a persona-grounded user simulator, a target model that separates intent perception from response generation, and an independent third-party LLM judge with optional multi-judge ensembling. Personas are sampled from public human-curated corpora, Nemotron-Personas-USA and PersonaMem-v2, rather than synthesized, reducing LLM-induced persona bias. EYT-Bench also introduces two trajectory-level metrics: embedding-based intent drift and final-intent completion rate (FICR), inspired by tau-bench. In a 17-target x 200-dialogue evaluation, EYT-Bench reveals four findings: (i) state-of-the-art closed- and open-source models are statistically close on subjective dimensions (empathy / persona / anthropomorphism vary within <= 0.3), but differ by up to 9x on objective intent tracking; (ii) reasoning ("thinking on") sharply improves objective tracking on long-context personas (+0.47-0.50 latent-intent accuracy on Gemma-4) while leaving subjective scores nearly unchanged; (iii) persona format dominates trajectory spread, with FICR saturating above 0.95 on Nemotron-USA but spreading from 0.53 to 0.88 on PersonaMem-v2; and (iv) the warm-up effect is robust on 16/17 models (one outlier, GPT-5.5, reverses the effect), with stable rankings across alpha in [0.05, 0.15]. A cross-judge ablation using deepseek-v4-pro confirms that target rankings and final-intent satisfaction are preserved across judges.

Related papers

• Creating an Atomic User Model for Personality-Aware Large Language Model Interaction [0.0]
  Atomic User Model (AUM) is a human-readable representation organising a person as a stable identity nucleus.<n>A pipeline where a task classifier, component-selection function and budgeted retriever return a small payload of fields at generation time.<n>Retrieving eight fields matched the style fidelity of the full user model on 23% of the context.
  arXiv  Detail & Related papers  (2026-09-10T18:11:43Z)
• Cognitive World Models for Process-Level Social Influence Evaluation [12.674232662629898]
  textbfCognitive textbfWorld textbfModel textbf(CogWM), an LLM-based user model.<n>CogWM moves social influence dialogue evaluation from terminal judgment to process tracking.
  arXiv  Detail & Related papers  (2026-06-28T16:49:43Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• MANTA: Multi-turn Assessment for Nonhuman Thinking & Alignment [0.1452394725421793]
  MANTA is a dynamic multi-turn evaluation framework built on the Inspect AI platform.<n>We present preliminary results from evaluations of claude-sonnet-4-20250514 and openai/gpt-4o.<n>We additionally present STYLEJUDGE, a controlled four-judge study demonstrating systematic format bias in LLM-as-judge evaluation.
  arXiv  Detail & Related papers  (2026-04-18T19:51:32Z)
• QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs [29.26861081722613]
  We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics.<n>We introduce QEDBench, the first large-scale dual-rubric alignment benchmark to measure alignment with human experts on university-level math.<n>We reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias.
  arXiv  Detail & Related papers  (2026-02-24T07:23:28Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Probabilistic Human Intent Prediction for Mobile Manipulation: An Evaluation with Human-Inspired Constraints [2.2893865000399938]
  Accurate inference of human intent enables human-robot collaboration without constraining human control or causing conflicts between humans and robots.<n>We present GUIDER, a probabilistic framework that enables a robot to estimate the intent of human operators.<n>We evaluated GUIDER on 25 trials (five participants x five task variants) in Isaac Sim, and compared it with two baselines, one for navigation and one for manipulation.
  arXiv  Detail & Related papers  (2025-07-14T10:21:27Z)
• Mind the Gap! Static and Interactive Evaluations of Large Audio Models [55.87220295533817]
  Large Audio Models (LAMs) are designed to power voice-native experiences.<n>This study introduces an interactive approach to evaluate LAMs and collect 7,500 LAM interactions from 484 participants.
  arXiv  Detail & Related papers  (2025-02-21T20:29:02Z)
• Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning [67.33899440998175]
  Chain-of-Opinion (COO) is a simple four-step solution modeling which and how to reason with personae.<n>COO distinguishes between explicit personae (demographics and ideology) and implicit personae (historical opinions)<n>COO efficiently achieves new state-of-the-art opinion prediction via prompting with only 5 inference calls, improving prior techniques by up to 4%.
  arXiv  Detail & Related papers  (2023-11-14T18:48:27Z)
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models [66.12432440863816]
  We propose Prometheus, a fully open-source Large Language Model (LLM) that is on par with GPT-4's evaluation capabilities. Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics. Prometheus achieves the highest accuracy on two human preference benchmarks.
  arXiv  Detail & Related papers  (2023-10-12T16:50:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.