Emergent Communication of Generalizations

• URL: http://arxiv.org/abs/2106.02668v1
• Date: Fri, 4 Jun 2021 19:02:18 GMT
• Title: Emergent Communication of Generalizations
• Authors: Jesse Mu, Noah Goodman
• Abstract summary: We argue that communicating about a single object in a shared visual context is prone to overfitting and does not encourage language useful beyond concrete reference. We propose games that require communicating generalizations over sets of objects representing abstract visual concepts. We find that these games greatly improve systematicity and interpretability of the learned languages.
• Score: 13.14792537601313
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To build agents that can collaborate effectively with others, recent research has trained artificial agents to communicate with each other in Lewis-style referential games. However, this often leads to successful but uninterpretable communication. We argue that this is due to the game objective: communicating about a single object in a shared visual context is prone to overfitting and does not encourage language useful beyond concrete reference. In contrast, human language conveys a rich variety of abstract ideas. To promote such skills, we propose games that require communicating generalizations over sets of objects representing abstract visual concepts, optionally with separate contexts for each agent. We find that these games greatly improve systematicity and interpretability of the learned languages, according to several metrics in the literature. Finally, we propose a method for identifying logical operations embedded in the emergent languages by learning an approximate compositional reconstruction of the language.

Related papers

• Visually Grounded Language Learning: a review of language games, datasets, tasks, and models [60.2604624857992]
  Many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field.
  arXiv  Detail & Related papers  (2023-12-05T02:17:29Z)
• EC^2: Emergent Communication for Embodied Control [72.99894347257268]
  Embodied control requires agents to leverage multi-modal pre-training to quickly learn how to act in new environments. We propose Emergent Communication for Embodied Control (EC2), a novel scheme to pre-train video-language representations for few-shot embodied control. EC2 is shown to consistently outperform previous contrastive learning methods for both videos and texts as task inputs.
  arXiv  Detail & Related papers  (2023-04-19T06:36:02Z)
• Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches [28.47300996711215]
  People rely heavily on context to enrich meaning beyond what is literally said. To interact successfully with people, user-facing artificial intelligence systems will require similar skills in pragmatics.
  arXiv  Detail & Related papers  (2022-11-15T18:21:46Z)
• Emergent Communication: Generalization and Overfitting in Lewis Games [53.35045559317384]
  Lewis signaling games are a class of simple communication games for simulating the emergence of language. In these games, two agents must agree on a communication protocol in order to solve a cooperative task. Previous work has shown that agents trained to play this game with reinforcement learning tend to develop languages that display undesirable properties.
  arXiv  Detail & Related papers  (2022-09-30T09:50:46Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Expressivity of Emergent Language is a Trade-off between Contextual Complexity and Unpredictability [7.765925231148388]
  We propose a definition of partial order between expressivity based on the generalisation performance across different language games. We also validate the hypothesis that expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context. We show that using our contrastive loss alleviates the collapse of message types seen using standard referential loss functions.
  arXiv  Detail & Related papers  (2021-06-07T21:57:11Z)
• Zero-Shot Generalization using Intrinsically Motivated Compositional Emergent Protocols [0.0]
  We show how compositionality can enable agents to not only interact with unseen objects but also transfer skills from one task to another in a zero-shot setting. We demonstrate how compositionality can enable agents to not only interact with unseen objects but also transfer skills from one task to another in a zero-shot setting.
  arXiv  Detail & Related papers  (2021-05-11T14:20:26Z)
• Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games [64.11746320061965]
  We study reinforcement learning for text-based games, which are interactive simulations in the context of natural language. We aim to conduct explicit reasoning with knowledge graphs for decision making, so that the actions of an agent are generated and supported by an interpretable inference procedure. We extensively evaluate our method on a number of man-made benchmark games, and the experimental results demonstrate that our method performs better than existing text-based agents.
  arXiv  Detail & Related papers  (2020-10-22T12:40:22Z)
• Incorporating Pragmatic Reasoning Communication into Emergent Language [38.134221799334426]
  We study the dynamics of linguistic communication along substantially different intelligence and intelligence levels. We propose computational models that combine short-term mutual reasoning-based pragmatics with long-term language emergentism. Our results shed light on their importance for making inroads towards getting more natural, accurate, robust, fine-grained, and succinct utterances.
  arXiv  Detail & Related papers  (2020-06-07T10:31:06Z)
• Emergence of Pragmatics from Referential Game between Theory of Mind Agents [64.25696237463397]
  We propose an algorithm, using which agents can spontaneously learn the ability to "read between lines" without any explicit hand-designed rules. We integrate the theory of mind (ToM) in a cooperative multi-agent pedagogical situation and propose an adaptive reinforcement learning (RL) algorithm to develop a communication protocol.
  arXiv  Detail & Related papers  (2020-01-21T19:37:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.