LVLMs are Bad at Overhearing Human Referential Communication

• URL: http://arxiv.org/abs/2509.11514v2
• Date: Fri, 24 Oct 2025 00:25:59 GMT
• Title: LVLMs are Bad at Overhearing Human Referential Communication
• Abstract summary: We study the capabilities of seven state-of-the-art Large Vision Language Models as overhearers to a corpus of spontaneous conversations.<n>We find that such a task remains challenging for current LVLMs and they all fail to show a consistent performance improvement.
• Score: 12.45317515017514
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: During spontaneous conversations, speakers collaborate on novel referring expressions, which they can then re-use in subsequent conversations. Understanding such referring expressions is an important ability for an embodied agent, so that it can carry out tasks in the real world. This requires integrating and understanding language, vision, and conversational interaction. We study the capabilities of seven state-of-the-art Large Vision Language Models (LVLMs) as overhearers to a corpus of spontaneous conversations between pairs of human discourse participants engaged in a collaborative object-matching task. We find that such a task remains challenging for current LVLMs and they all fail to show a consistent performance improvement as they overhear more conversations from the same discourse participants repeating the same task for multiple rounds. We release our corpus and code for reproducibility and to facilitate future research.

Related papers

• Multimodal Conversation Structure Understanding [12.29827265137757]
  Large language models' ability to understand fine-grained conversational structure remains underexplored.<n>We present a human annotated dataset of 4,398 annotations for speakers and reply-to relationship, 5,755 addressees, and 3,142 side-participants.<n>We evaluate popular audio-visual LLMs and vision-language models on our dataset, and our experimental results suggest that multimodal conversational structure understanding remains challenging.
  arXiv  Detail & Related papers  (2025-05-23T06:41:54Z)
• Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data [33.85748258158527]
  Empathetic dialogue is crucial for natural human-computer interaction.<n>Large language models (LLMs) have revolutionized dialogue generation by harnessing their powerful capabilities.<n>We propose a novel approach that circumvents the need for question-answering data.
  arXiv  Detail & Related papers  (2025-01-19T04:10:53Z)
• Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations [58.65755268815283]
  Many real dialogues are interactive, meaning an agent's utterances will influence their conversational partner, elicit information, or change their opinion. We use this fact to rewrite and augment existing suboptimal data, and train via offline reinforcement learning (RL) an agent that outperforms both prompting and learning from unaltered human demonstrations. Our results in a user study with real humans show that our approach greatly outperforms existing state-of-the-art dialogue agents.
  arXiv  Detail & Related papers  (2024-11-07T21:37:51Z)
• Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models [56.93074140619464]
  We propose RiC (Reasoning in Conversation), a method that focuses on solving subjective tasks through dialogue simulation. The motivation of RiC is to mine useful contextual information by simulating dialogues instead of supplying chain-of-thought style rationales. We evaluate both API-based and open-source LLMs including GPT-4, ChatGPT, and OpenChat across twelve tasks.
  arXiv  Detail & Related papers  (2024-02-27T05:37:10Z)
• Sibyl: Empowering Empathetic Dialogue Generation in Large Language Models via Sensible and Visionary Commonsense Inference [40.96005200292604]
  We present an innovative framework named Sensible and Visionary Commonsense Knowledge (Sibyl)<n>It is designed to concentrate on the immediately succeeding dialogue, aiming to elicit more empathetic responses.<n> Experimental results demonstrate that incorporating our paradigm for acquiring commonsense knowledge into LLMs comprehensively enhances the quality of their responses.
  arXiv  Detail & Related papers  (2023-11-26T14:35:23Z)
• Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue [73.69510478736483]
  Large language models (LLMs) can generate fluent, coherent, and diverse responses. However, they lack a crucial ability: communication skills. This article aims to empower LLMs with communication skills through inner monologues. Experimental results show that the proposed CSIM strategy improves the backbone models and outperforms the baselines.
  arXiv  Detail & Related papers  (2023-11-13T16:19:42Z)
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations [70.7884839812069]
  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue.
  arXiv  Detail & Related papers  (2023-11-09T18:45:16Z)
• Interactive Conversational Head Generation [68.76774230274076]
  We introduce a new conversation head generation benchmark for synthesizing behaviors of a single interlocutor in a face-to-face conversation. The capability to automatically synthesize interlocutors which can participate in long and multi-turn conversations is vital and offer benefits for various applications.
  arXiv  Detail & Related papers  (2023-07-05T08:06:26Z)
• Few-shot Language Coordination by Modeling Theory of Mind [95.54446989205117]
  We study the task of few-shot $textitlanguage coordination$. We require the lead agent to coordinate with a $textitpopulation$ of agents with different linguistic abilities. This requires the ability to model the partner's beliefs, a vital component of human communication.
  arXiv  Detail & Related papers  (2021-07-12T19:26:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.