Human-robot conversation with multiple participants in noisy public spaces

• URL: http://arxiv.org/abs/2609.00648v1
• Date: Tue, 01 Sep 2026 03:25:03 GMT
• Title: Human-robot conversation with multiple participants in noisy public spaces
• Abstract summary: This work proposes an audio system that can be used for speech recognition and transmission as clean speech to a remote operator.<n>The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots.<n>We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.
• Score: 15.265544600480068
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For noisy real-world environments such as those in open public spaces, spoken dialogue systems for both autonomous robots and avatars should be carefully designed to provide enhanced speech signals. These signals can be used either for speech recognition or, in the case of an avatar system, transmitted as clean speech to a remote operator. This work proposes an audio system that can be used for both these scenarios and was demonstrated as a proof-of-concept at the 2025 World Expo in Osaka. The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots, with one of them acting as an avatar for a remote operator. Both systems feature multi-party conversation and use a single multi-channel microphone array. We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.

Related papers

• KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction [29.52851418305386]
  KABURITTS takes a per-speaker phoneme as input and renders the speech of two speakers on per-frame phonemes.<n>A user evaluation shows that, compared with strong baselines, the proposed method attains naturalness at both utterance and the interaction level.
  arXiv  Detail & Related papers  (2026-09-07T08:24:10Z)
• Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO [18.561388308227556]
  We introduce Cocktail-Talker, a speech framework for multi-speaker spoken dialog modeling in noisy social environments.<n> Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token.<n>We also develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings.
  arXiv  Detail & Related papers  (2026-07-30T06:53:15Z)
• F-Actor: Controllable Conversational Behaviour in Full-Duplex Models [70.48189107402145]
  We present first open, instruction-following full-stage conversational speech model that can be trained efficiently under typical academic resource constraints.<n>Our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage pretraining.<n>Both the model and training code will be released to enable reproducible research on controllable full-like controllable full-stage speech systems.
  arXiv  Detail & Related papers  (2026-01-16T14:25:57Z)
• TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation [72.46711449668814]
  We introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner.<n>We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction, and speech quality.
  arXiv  Detail & Related papers  (2025-12-23T12:04:23Z)
• Moshi: a speech-text foundation model for real-time dialogue [78.88479749811376]
  Current systems for spoken dialogue rely on pipelines independent voice activity detection and text-to-speech. We show how Moshi Moshi can provide streaming speech recognition and text-to-speech. Our resulting model is first real-time full spoken large language model modality.
  arXiv  Detail & Related papers  (2024-09-17T17:55:39Z)
• Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation [55.043492250775294]
  We introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response. We also introduce MultiDialog, the first large-scale multimodal spoken dialogue corpus.
  arXiv  Detail & Related papers  (2024-06-12T04:48:36Z)
• AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation [58.72068260933836]
  The input and output of the system are multimodal (i.e., audio and visual speech) We can perform real-like conversations with individuals worldwide in a virtual meeting by utilizing our own primary languages. In contrast to Speech-to-Speech Translation (A2A), which solely translates between audio modalities, the proposed AV2AV directly translates between audio-visual speech.
  arXiv  Detail & Related papers  (2023-12-05T05:36:44Z)
• A Real-time Speaker Diarization System Based on Spatial Spectrum [14.189768987932364]
  We propose a novel systematic approach to tackle several long-standing challenges in speaker diarization tasks. First, a differential directional microphone array-based approach is exploited to capture the target speakers' voice in far-field adverse environment. Second, an online speaker-location joint clustering approach is proposed to keep track of speaker location. Third, an instant speaker number detector is developed to trigger the mechanism that separates overlapped speech.
  arXiv  Detail & Related papers  (2021-07-20T08:25:23Z)
• Self-supervised reinforcement learning for speaker localisation with the iCub humanoid robot [58.2026611111328]
  Looking at a person's face is one of the mechanisms that humans rely on when it comes to filtering speech in noisy environments. Having a robot that can look toward a speaker could benefit ASR performance in challenging environments. We propose a self-supervised reinforcement learning-based framework inspired by the early development of humans.
  arXiv  Detail & Related papers  (2020-11-12T18:02:15Z)
• I can attend a meeting too! Towards a human-like telepresence avatar robot to attend meeting on your behalf [8.512048419752047]
  We focus on a telepresence robot that can be used to attend a meeting remotely with a group of people. To provide a better meeting experience, the robot should localize the speaker and bring the speaker at the center of the viewing angle. This article presents a study and implementation of an attention shifting scheme in a telepresence meeting scenario.
  arXiv  Detail & Related papers  (2020-06-28T16:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.