Learning to Draw: Emergent Communication through Sketching

• URL: http://arxiv.org/abs/2106.02067v1
• Date: Thu, 3 Jun 2021 18:17:55 GMT
• Title: Learning to Draw: Emergent Communication through Sketching
• Authors: Daniela Mihai, Jonathon Hare
• Abstract summary: We show how agents can learn to communicate in order to collaboratively solve tasks. Existing research has focused on language, with a learned communication channel transmitting sequences of discrete tokens between the agents. Our agents are parameterised by deep neural networks, and the drawing procedure is differentiable, allowing for end-to-end training. In the framework of a referential communication game, we demonstrate that agents can not only successfully learn to communicate by drawing, but with appropriate inductive biases, can do so in a fashion that humans can interpret.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evidence that visual communication preceded written language and provided a basis for it goes back to prehistory, in forms such as cave and rock paintings depicting traces of our distant ancestors. Emergent communication research has sought to explore how agents can learn to communicate in order to collaboratively solve tasks. Existing research has focused on language, with a learned communication channel transmitting sequences of discrete tokens between the agents. In this work, we explore a visual communication channel between agents that are allowed to draw with simple strokes. Our agents are parameterised by deep neural networks, and the drawing procedure is differentiable, allowing for end-to-end training. In the framework of a referential communication game, we demonstrate that agents can not only successfully learn to communicate by drawing, but with appropriate inductive biases, can do so in a fashion that humans can interpret. We hope to encourage future research to consider visual communication as a more flexible and directly interpretable alternative of training collaborative agents.

Related papers

• Less Data, More Knowledge: Building Next Generation Semantic Communication Networks [180.82142885410238]
  We present the first rigorous vision of a scalable end-to-end semantic communication network. We first discuss how the design of semantic communication networks requires a move from data-driven networks towards knowledge-driven ones. By using semantic representation and languages, we show that the traditional transmitter and receiver now become a teacher and apprentice.
  arXiv  Detail & Related papers  (2022-11-25T19:03:25Z)
• Emergent Graphical Conventions in a Visual Communication Game [80.79297387339614]
  Humans communicate with graphical sketches apart from symbolic languages. We take the very first step to model and simulate such an evolution process via two neural agents playing a visual communication game. We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions.
  arXiv  Detail & Related papers  (2021-11-28T18:59:57Z)
• Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent? [0.0]
  We show that a combination of powerful pretrained encoder networks, with appropriate inductive biases, can lead to agents that draw recognisable sketches. We develop an approach to help automatically analyse the semantic content being conveyed by a sketch.
  arXiv  Detail & Related papers  (2021-10-15T17:02:34Z)
• Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents [83.52684405389445]
  We introduce the collaborative multi-object navigation task CoMON. In this task, an oracle agent has detailed environment information in the form of a map. It communicates with a navigator agent that perceives the environment visually and is tasked to find a sequence of goals. We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment.
  arXiv  Detail & Related papers  (2021-10-12T06:56:11Z)
• Few-shot Language Coordination by Modeling Theory of Mind [95.54446989205117]
  We study the task of few-shot $textitlanguage coordination$. We require the lead agent to coordinate with a $textitpopulation$ of agents with different linguistic abilities. This requires the ability to model the partner's beliefs, a vital component of human communication.
  arXiv  Detail & Related papers  (2021-07-12T19:26:11Z)
• Interpretable agent communication from scratch(with a generic visual processor emerging on the side) [29.722833768572805]
  We train two deep nets from scratch to perform realistic referent identification through unsupervised emergent communication. We show that the largely interpretable emergent protocol allows the nets to successfully communicate even about object types they did not see at training time. Our results provide concrete evidence of the viability of (interpretable) emergent deep net communication in a more realistic scenario than previously considered.
  arXiv  Detail & Related papers  (2021-06-08T11:32:11Z)
• The emergence of visual semantics through communication games [0.0]
  Communication systems which capture visual semantics can be learned in a completely self-supervised manner by playing the right types of game. Our work bridges a gap between emergent communication research and self-supervised feature learning.
  arXiv  Detail & Related papers  (2021-01-25T17:43:37Z)
• Learning to cooperate: Emergent communication in multi-agent navigation [49.11609702016523]
  We show that agents performing a cooperative navigation task learn an interpretable communication protocol. An analysis of the agents' policies reveals that emergent signals spatially cluster the state space. Using populations of agents, we show that the emergent protocol has basic compositional structure.
  arXiv  Detail & Related papers  (2020-04-02T16:03:17Z)
• Emergence of Pragmatics from Referential Game between Theory of Mind Agents [64.25696237463397]
  We propose an algorithm, using which agents can spontaneously learn the ability to "read between lines" without any explicit hand-designed rules. We integrate the theory of mind (ToM) in a cooperative multi-agent pedagogical situation and propose an adaptive reinforcement learning (RL) algorithm to develop a communication protocol.
  arXiv  Detail & Related papers  (2020-01-21T19:37:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.