Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channel

• URL: http://arxiv.org/abs/2111.02827v2
• Date: Tue, 2 May 2023 10:11:25 GMT
• Title: Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channel
• Authors: Kevin Eloff, Okko R\"as\"anen, Herman A. Engelbrecht, Arnu Pretorius, Herman Kamper
• Abstract summary: We ask: Are we able to observe emergent language between agents with a continuous communication channel? We propose a messaging environment where a Speaker agent needs to convey a set of attributes to a Listener over a noisy acoustic channel. Using DQN to train our agents, we show that: (1) unlike the discrete case, the acoustic Speaker learns redundancy to improve Listener coherency, (2) the acoustic Speaker develops more compositional communication protocols which implicitly compensates for transmission errors over a noisy channel, and (3) DQN has significant performance gains and increased compositionality when compared to previous methods optimised using REINFORCE.
• Score: 21.503787009047677
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-agent reinforcement learning has been used as an effective means to study emergent communication between agents, yet little focus has been given to continuous acoustic communication. This would be more akin to human language acquisition; human infants acquire language in large part through continuous signalling with their caregivers. We therefore ask: Are we able to observe emergent language between agents with a continuous communication channel? Our goal is to provide a platform to begin bridging the gap between human and agent communication, allowing us to analyse continuous signals, how they emerge, their characteristics, and how they relate to human language acquisition. We propose a messaging environment where a Speaker agent needs to convey a set of attributes to a Listener over a noisy acoustic channel. Using DQN to train our agents, we show that: (1) unlike the discrete case, the acoustic Speaker learns redundancy to improve Listener coherency, (2) the acoustic Speaker develops more compositional communication protocols which implicitly compensates for transmission errors over a noisy channel, and (3) DQN has significant performance gains and increased compositionality when compared to previous methods optimised using REINFORCE.

Related papers

• Language-Oriented Communication with Semantic Coding and Knowledge Distillation for Text-to-Image Generation [53.97155730116369]
  We put forward a novel framework of language-oriented semantic communication (LSC) In LSC, machines communicate using human language messages that can be interpreted and manipulated via natural language processing (NLP) techniques for SC efficiency. We introduce three innovative algorithms: 1) semantic source coding (SSC), which compresses a text prompt into its key head words capturing the prompt's syntactic essence; 2) semantic channel coding ( SCC), that improves robustness against errors by substituting head words with their lenghthier synonyms; and 3) semantic knowledge distillation (SKD), that produces listener-customized prompts via in-context learning the listener's
  arXiv  Detail & Related papers  (2023-09-20T08:19:05Z)
• Improving Speaker Diarization using Semantic Information: Joint Pairwise Constraints Propagation [53.01238689626378]
  We propose a novel approach to leverage semantic information in speaker diarization systems. We introduce spoken language understanding modules to extract speaker-related semantic information. We present a novel framework to integrate these constraints into the speaker diarization pipeline.
  arXiv  Detail & Related papers  (2023-09-19T09:13:30Z)
• Emergent Quantized Communication [34.31732248872158]
  We propose an alternative approach to achieve discrete communication -- quantization of communicated messages. Message quantization allows us to train the model end-to-end, achieving superior performance in multiple setups.
  arXiv  Detail & Related papers  (2022-11-04T12:39:45Z)
• Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents [83.52684405389445]
  We introduce the collaborative multi-object navigation task CoMON. In this task, an oracle agent has detailed environment information in the form of a map. It communicates with a navigator agent that perceives the environment visually and is tasked to find a sequence of goals. We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment.
  arXiv  Detail & Related papers  (2021-10-12T06:56:11Z)
• Emergent Discrete Communication in Semantic Spaces [3.2280079436668996]
  We propose a neural agent architecture that enables agents to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimize communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments.
  arXiv  Detail & Related papers  (2021-08-04T03:32:48Z)
• Few-shot Language Coordination by Modeling Theory of Mind [95.54446989205117]
  We study the task of few-shot $textitlanguage coordination$. We require the lead agent to coordinate with a $textitpopulation$ of agents with different linguistic abilities. This requires the ability to model the partner's beliefs, a vital component of human communication.
  arXiv  Detail & Related papers  (2021-07-12T19:26:11Z)
• Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning [36.468498804251574]
  We show that discrete message communication has performance comparable to continuous message communication. We propose an approach that allows humans to interactively send discrete messages to agents.
  arXiv  Detail & Related papers  (2021-02-24T20:44:14Z)
• Effective Communications: A Joint Learning and Communication Framework for Multi-Agent Reinforcement Learning over Noisy Channels [0.0]
  We propose a novel formulation of the "effectiveness problem" in communications. We consider multiple agents communicating over a noisy channel in order to achieve better coordination and cooperation. We show via examples that the joint policy learned using the proposed framework is superior to that where the communication is considered separately.
  arXiv  Detail & Related papers  (2021-01-02T10:43:41Z)
• Augmentation adversarial training for self-supervised speaker recognition [49.47756927090593]
  We train robust speaker recognition models without speaker labels. Experiments on VoxCeleb and VOiCES datasets show significant improvements over previous works using self-supervision.
  arXiv  Detail & Related papers  (2020-07-23T15:49:52Z)
• Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention [70.82604384963679]
  This paper investigates a self-adaptation method for speech enhancement using auxiliary speaker-aware features. We extract a speaker representation used for adaptation directly from the test utterance.
  arXiv  Detail & Related papers  (2020-02-14T05:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.