Defending Compositionality in Emergent Languages

• URL: http://arxiv.org/abs/2206.04751v1
• Date: Thu, 9 Jun 2022 20:13:46 GMT
• Title: Defending Compositionality in Emergent Languages
• Authors: Michal Auersperger, Pavel Pecina
• Abstract summary: We argue that some conclusions are too strong and/or incomplete. In the context of a two-agent communication game, we show that compositionality indeed seems essential for successful generalization when the evaluation is done on a proper dataset.
• Score: 3.04585143845864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compositionality has traditionally been understood as a major factor in productivity of language and, more broadly, human cognition. Yet, recently, some research started to question its status, showing that artificial neural networks are good at generalization even without noticeable compositional behavior. We argue that some of these conclusions are too strong and/or incomplete. In the context of a two-agent communication game, we show that compositionality indeed seems essential for successful generalization when the evaluation is done on a proper dataset.

Related papers

• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models [64.43764443000003]
  We identify two sources of visual-linguistic compositionality: linguistic priors and the interplay between images and texts. We propose a new metric for compositionality without such linguistic priors.
  arXiv  Detail & Related papers  (2023-10-04T12:48:33Z)
• On Evaluating Multilingual Compositional Generalization with Translated Datasets [34.51457321680049]
  We show that compositional generalization abilities differ across languages. We craft a faithful rule-based translation of the MCWQ dataset from English to Chinese and Japanese. Even with the resulting robust benchmark, which we call MCWQ-R, we show that the distribution of compositions still suffers due to linguistic divergences.
  arXiv  Detail & Related papers  (2023-06-20T10:03:57Z)
• DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning [89.92601337474954]
  Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations. We introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding.
  arXiv  Detail & Related papers  (2023-06-15T10:41:23Z)
• How Do In-Context Examples Affect Compositional Generalization? [86.57079616209474]
  In this paper, we present CoFe, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples. Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple.
  arXiv  Detail & Related papers  (2023-05-08T16:32:18Z)
• The paradox of the compositionality of natural language: a neural machine translation case study [15.37696298313134]
  We re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation (NMT) The results highlight two main issues: the inconsistent behaviour of NMT models and their inability to (correctly) modulate between local and global processing.
  arXiv  Detail & Related papers  (2021-08-12T17:57:23Z)
• Compositional Processing Emerges in Neural Networks Solving Math Problems [100.80518350845668]
  Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed. Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
  arXiv  Detail & Related papers  (2021-05-19T07:24:42Z)
• On (Emergent) Systematic Generalisation and Compositionality in Visual Referential Games with Straight-Through Gumbel-Softmax Estimator [0.30458514384586394]
  The drivers of compositionality emerge when two (or more) agents play a non-visual referential game. This paper investigates what extent the drivers of compositionality identified so far in the field apply in the ST-GS context. Using the ST-GS approach with small batch sizes and an overcomplete communication channel improves compositionality in the emerging languages.
  arXiv  Detail & Related papers  (2020-12-19T20:40:09Z)
• Emergent Language Generalization and Acquisition Speed are not tied to Compositionality [31.6793931695019]
  We argue that these beneficial properties are only loosely connected to compositionality. In two experiments, we demonstrate that, depending on the task, non-compositional languages might show equal, or better, generalization performance and acquisition speed.
  arXiv  Detail & Related papers  (2020-04-07T14:10:27Z)
• A Benchmark for Systematic Generalization in Grounded Language Understanding [61.432407738682635]
  Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts. Modern neural networks, by contrast, struggle to interpret novel compositions. We introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding.
  arXiv  Detail & Related papers  (2020-03-11T08:40:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.