Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation

• URL: http://arxiv.org/abs/2606.07541v1
• Date: Fri, 01 May 2026 05:55:45 GMT
• Title: Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation
• Abstract summary: Multimodal large language models (MLLMs) have shown strong performance on objective tasks such as video understanding and reasoning.<n>We evaluate MLLMs as synthetic participants in an emerging task: assessing perceived sensory engagement with short videos.<n>We find that even leading MLLMs show limited agreement with human participants.
• Score: 11.282431394902241
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) have shown strong performance on objective tasks such as video understanding and reasoning. However, it remains unclear whether they can approximate subjective human responses, which depend not only on content comprehension but also on individuals' social contexts. To address this gap, we evaluate MLLMs as synthetic participants in an emerging task: assessing perceived sensory engagement with short videos. Grounded in the Perceived Message Sensation Value (PMSV) framework, we compare ratings from recruited human participants and profile-conditioned MLLM simulations (n=673) using a 17-item scale measuring emotional arousal, dramatic impact, and novelty. We find that even leading MLLMs (Gemini 3 Flash and Qwen 3 Omni) show limited agreement with human participants. The models exhibit distinct downward mean-shift and central-tendency biases in their rating distributions. They both introduce and flatten subgroup differences, while showing inconsistent sensitivity to participant profiles. Prompting strategies affect these metrics differently, modestly improving some aspects while worsening others. These results highlight both the challenges and opportunities of developing MLLMs as synthetic participants in video-based research. Data and code: https://github.com/MINDLab25/mllm-human-simulation-eval

Related papers

• Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments [10.717755637740948]
  Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations.<n>We conduct a systematic comparison using a naturally occurring online persuasion corpus.<n>Our results show that LLMs achieve only slight agreement with humans.
  arXiv  Detail & Related papers  (2026-08-30T14:08:06Z)
• Emulating Aggregate Human Choice Behavior and Biases with GPT Conversational Agents [0.48439699124726004]
  Large language models (LLMs) have been shown to reproduce well-known biases.<n>We adapted three well-established decision scenarios into a conversational setting and conducted a human experiment.<n>We found notable differences between models in how they aligned human behavior.
  arXiv  Detail & Related papers  (2026-02-05T12:33:05Z)
• Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies [54.08697738311866]
  Social deduction games like Werewolf combine language, reasoning, and strategy.<n>We curate a high-quality, human-verified multimodal Werewolf dataset containing over 100 hours of video, 32.4M utterance tokens, and 15 rule variants.<n>We propose a novel strategy-alignment evaluation that leverages the winning faction's strategies as ground truth in two stages.
  arXiv  Detail & Related papers  (2025-10-13T13:33:30Z)
• Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach [29.502292089901825]
  We argue that this inconsistency stems partly from constraints in existing evaluation methods.<n>We propose an Emotion Statement Judgment task that overcomes these constraints.<n>We devise an automated pipeline that efficiently constructs emotion-centric statements with minimal human effort.
  arXiv  Detail & Related papers  (2025-09-26T06:30:39Z)
• Large Language Models are Highly Aligned with Human Ratings of Emotional Stimuli [0.62914438169038]
  Emotions exert an immense influence over human behavior and cognition in both commonplace and high-stress tasks.<n>Discussions should be informed by an understanding of how large language models evaluate emotionally loaded stimuli or situations.<n>A model's alignment with human behavior in these cases can inform the effectiveness of LLMs for certain roles or interactions.
  arXiv  Detail & Related papers  (2025-08-19T19:22:00Z)
• Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? [81.49470136653665]
  We evaluate the robustness and expressiveness of value representations across three widely used probing strategies.<n>We show that the demographic context has little effect on the free-text generation, and the models' values only weakly correlate with their preference for value-based actions.
  arXiv  Detail & Related papers  (2025-07-17T18:56:41Z)
• HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding [120.84817886550765]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks involving both images and videos.<n>Existing human-centric benchmarks predominantly emphasize video generation quality and action recognition, while overlooking essential perceptual and cognitive abilities required in human-centered scenarios.<n>We propose a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric video understanding.
  arXiv  Detail & Related papers  (2025-07-07T11:52:24Z)
• Bridging the Gap: In-Context Learning for Modeling Human Disagreement [8.011316959982654]
  Large Language Models (LLMs) have shown strong performance on NLP classification tasks.<n>This study examines whether LLMs can capture multiple perspectives and reflect annotator disagreement in subjective tasks such as hate speech and offensive language detection.
  arXiv  Detail & Related papers  (2025-06-06T14:24:29Z)
• Evaluating Graphical Perception with Multimodal LLMs [2.090547583226381]
  Multimodal Large Language Models (MLLMs) have remarkably progressed in analyzing and understanding images.<n>For visualization, how do MLLMs perform when applied to graphical perception tasks?<n>Our study primarily evaluates fine-tuned and pretrained models and zero-shot prompting to determine if they closely match human graphical perception.
  arXiv  Detail & Related papers  (2025-04-05T16:14:08Z)
• If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs [55.8331366739144]
  We introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in large language models (LLMs)<n>Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches.
  arXiv  Detail & Related papers  (2025-03-30T16:50:57Z)
• HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data [55.739633494946204]
  We present HumanVBench, an innovative benchmark meticulously crafted to bridge gaps in the evaluation of video MLLMs.<n>HumanVBench comprises 16 carefully designed tasks that explore two primary dimensions: inner emotion and outer manifestations, spanning static and dynamic, basic and complex, as well as single-modal and cross-modal aspects.<n>A comprehensive evaluation across 22 SOTA video MLLMs reveals notable limitations in current performance, especially in cross-modal and emotion perception.
  arXiv  Detail & Related papers  (2024-12-23T13:45:56Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)
• Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench [83.41621219298489]
  We evaluate Large Language Models' (LLMs) anthropomorphic capabilities using the emotion appraisal theory from psychology. We collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. We conduct a human evaluation involving more than 1,200 subjects worldwide.
  arXiv  Detail & Related papers  (2023-08-07T15:18:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.