Learning to Refer from Estimated Listener Gaze

• URL: http://arxiv.org/abs/2609.14207v1
• Date: Sun, 13 Sep 2026 00:36:18 GMT
• Title: Learning to Refer from Estimated Listener Gaze
• Abstract summary: We propose to finetune vision-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental listener comprehension, in the form of gaze scanpaths, into learning signals.<n>We find that speaker policies trained with gaze-estimating listeners result in significantly more pragmatically-optimal references than base models, reducing sequence length from 15.4 down to 4.0 words while increasing referential success from 75.2 up to 80.0%.
• Score: 12.172774856748347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose to finetune vision-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental listener comprehension, in the form of gaze scanpaths, into learning signals. During training, referring expressions are sampled from the speaker policy being optimized, conditioned on images and target referents; then, a neural listener estimating human gaze behavior maps from images and sampled referring expressions to scanpaths, each represented by a sequence of fixations, with each fixation corresponding to a word in the referring expression. We experiment with several approaches to convert fixation sequences and target referents into token- and sequence-level rewards, which are used to optimize policy parameters. Through evaluation with human listeners, we find that speaker policies trained with gaze-estimating listeners result in significantly more pragmatically-optimal references than base models, reducing sequence length from 15.4 down to 4.0 words while increasing referential success from 75.2 up to 80.0%. Our work demonstrates a promising opportunity for learning to generate utterances through language-based interaction, not only from the explicit signal of communicative success, but also from implicitly-available observations of a listener's process of comprehension.

Related papers

• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Entropy-based Coarse and Compressed Semantic Speech Representation Learning [72.18542411704347]
  We propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations.<n> Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences.
  arXiv  Detail & Related papers  (2025-08-30T13:50:58Z)
• Grounding Language in Multi-Perspective Referential Communication [16.421832484760987]
  We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments. We collect a dataset of 2,970 human-written referring expressions, each paired with human comprehension judgments. We evaluate the performance of automated models as speakers and listeners paired with human partners, finding that model performance in both reference generation and comprehension lags behind that of pairs of human agents.
  arXiv  Detail & Related papers  (2024-10-04T22:42:30Z)
• Speechworthy Instruction-tuned Language Models [71.8586707840169]
  We show that both prompting and preference learning increase the speech-suitability of popular instruction-tuned LLMs. We share lexical, syntactical, and qualitative analyses to showcase how each method contributes to improving the speech-suitability of generated responses.
  arXiv  Detail & Related papers  (2024-09-23T02:34:42Z)
• Leveraging WaveNet for Dynamic Listening Head Modeling from Speech [11.016004057765185]
  The creation of listener facial responses aims to simulate interactive communication feedback from a listener during a face-to-face conversation. Our approach focuses on capturing the subtle nuances of listener feedback, ensuring the preservation of individual listener identity.
  arXiv  Detail & Related papers  (2024-09-08T13:19:22Z)
• Can Language Models Learn to Listen? [96.01685069483025]
  We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study.
  arXiv  Detail & Related papers  (2023-08-21T17:59:02Z)
• Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering [78.2927924732142]
  We propose speaker-invariant clustering (Spin) as a novel self-supervised learning method. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU.
  arXiv  Detail & Related papers  (2023-05-18T15:59:36Z)
• Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning [15.729812221628382]
  We introduce a simple neural encoder architecture that can be trained using an unsupervised contrastive learning objective. We show that when built on top of recent self-supervised audio representations, this method can be applied iteratively and yield competitive SSE.
  arXiv  Detail & Related papers  (2022-04-11T14:28:01Z)
• LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech [67.88748572167309]
  We present LDNet, a unified framework for mean opinion score (MOS) prediction. We propose two inference methods that provide more stable results and efficient computation.
  arXiv  Detail & Related papers  (2021-10-18T08:52:31Z)
• Evaluating Models of Robust Word Recognition with Serial Reproduction [8.17947290421835]
  We compare several broad-coverage probabilistic generative language models in their ability to capture human linguistic expectations. We find that those models that make use of abstract representations of preceding linguistic context best predict the changes made by people in the course of serial reproduction.
  arXiv  Detail & Related papers  (2021-01-24T20:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.