Visual Referential Games Further the Emergence of Disentangled Representations

• URL: http://arxiv.org/abs/2304.14511v1
• Date: Thu, 27 Apr 2023 20:00:51 GMT
• Title: Visual Referential Games Further the Emergence of Disentangled Representations
• Authors: Kevin Denamgana\"i, Sondess Missaoui and James Alfred Walker
• Abstract summary: This paper investigates how do compositionality at the level of emerging languages, disentanglement at the level of the learned representations, and systematicity relate to each other in the context of visual referential games.
• Score: 0.12891210250935145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural languages are powerful tools wielded by human beings to communicate information. Among their desirable properties, compositionality has been the main focus in the context of referential games and variants, as it promises to enable greater systematicity to the agents which would wield it. The concept of disentanglement has been shown to be of paramount importance to learned representations that generalise well in deep learning, and is thought to be a necessary condition to enable systematicity. Thus, this paper investigates how do compositionality at the level of the emerging languages, disentanglement at the level of the learned representations, and systematicity relate to each other in the context of visual referential games. Firstly, we find that visual referential games that are based on the Obverter architecture outperforms state-of-the-art unsupervised learning approach in terms of many major disentanglement metrics. Secondly, we expand the previously proposed Positional Disentanglement (PosDis) metric for compositionality to (re-)incorporate some concerns pertaining to informativeness and completeness features found in the Mutual Information Gap (MIG) disentanglement metric it stems from. This extension allows for further discrimination between the different kind of compositional languages that emerge in the context of Obverter-based referential games, in a way that neither the referential game accuracy nor previous metrics were able to capture. Finally we investigate whether the resulting (emergent) systematicity, as measured by zero-shot compositional learning tests, correlates with any of the disentanglement and compositionality metrics proposed so far. Throughout the training process, statically significant correlation coefficients can be found both positive and negative depending on the moment of the measure.

Related papers

• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• Emergent Communication for Rules Reasoning [38.24159397787027]
  We propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules. Experimental results show that, in the Reasoning Game, a semantically stable and compositional language emerges to solve reasoning problems.
  arXiv  Detail & Related papers  (2023-11-08T05:57:39Z)
• Im-Promptu: In-Context Composition from Image Prompts [10.079743487034762]
  We investigate whether analogical reasoning can enable in-context composition over composable elements of visual stimuli. We use Im-Promptu to train agents with different levels of compositionality, including vector representations, patch representations, and object slots. Our experiments reveal tradeoffs between extrapolation abilities and the degree of compositionality, with non-compositional representations extending learned composition rules to unseen domains but performing poorly on tasks.
  arXiv  Detail & Related papers  (2023-05-26T21:10:11Z)
• Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
  Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning. We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
  arXiv  Detail & Related papers  (2023-05-19T08:22:23Z)
• Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding [143.5927158318524]
  Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. We introduce a new Compositional Temporal Grounding task and construct two new dataset splits. We argue that the inherent structured semantics inside the videos and language is the crucial factor to achieve compositional generalization.
  arXiv  Detail & Related papers  (2023-01-22T08:02:23Z)
• Learning Attention-based Representations from Multiple Patterns for Relation Prediction in Knowledge Graphs [2.4028383570062606]
  AEMP is a novel model for learning contextualized representations by acquiring entities' context information. AEMP either outperforms or competes with state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-07T10:53:35Z)
• On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations [74.70957445600936]
  Multiple metrics have been introduced to measure fairness in various natural language processing tasks. These metrics can be roughly categorized into two categories: 1) emphextrinsic metrics for evaluating fairness in downstream applications and 2) emphintrinsic metrics for estimating fairness in upstream language representation models.
  arXiv  Detail & Related papers  (2022-03-25T22:17:43Z)
• Integrating Language Guidance into Vision-based Deep Metric Learning [78.18860829585182]
  We propose to learn metric spaces which encode semantic similarities as embedding space. These spaces should be transferable to classes beyond those seen during training. This causes learned embedding spaces to encode incomplete semantic context and misrepresent the semantic relation between classes.
  arXiv  Detail & Related papers  (2022-03-16T11:06:50Z)
• On the Interplay Between Fine-tuning and Composition in Transformers [7.513100214864645]
  We investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information. Specifically, we fine-tune models on an adversarial paraphrase classification task with high lexical overlap, and on a sentiment classification task. We find that fine-tuning largely fails to benefit compositionality in these representations, though training on sentiment yields a small, localized benefit for certain models.
  arXiv  Detail & Related papers  (2021-05-31T01:49:56Z)
• On (Emergent) Systematic Generalisation and Compositionality in Visual Referential Games with Straight-Through Gumbel-Softmax Estimator [0.30458514384586394]
  The drivers of compositionality emerge when two (or more) agents play a non-visual referential game. This paper investigates what extent the drivers of compositionality identified so far in the field apply in the ST-GS context. Using the ST-GS approach with small batch sizes and an overcomplete communication channel improves compositionality in the emerging languages.
  arXiv  Detail & Related papers  (2020-12-19T20:40:09Z)
• Dynamic Semantic Matching and Aggregation Network for Few-shot Intent Detection [69.2370349274216]
  Few-shot Intent Detection is challenging due to the scarcity of available annotated utterances. Semantic components are distilled from utterances via multi-head self-attention. Our method provides a comprehensive matching measure to enhance representations of both labeled and unlabeled instances.
  arXiv  Detail & Related papers  (2020-10-06T05:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.