Development of Compositionality and Generalization through Interactive Learning of Language and Action of Robots

• URL: http://arxiv.org/abs/2403.19995v2
• Date: Tue, 23 Jul 2024 05:21:44 GMT
• Title: Development of Compositionality and Generalization through Interactive Learning of Language and Action of Robots
• Authors: Prasanna Vijayaraghavan, Jeffrey Frederic Queisser, Sergio Verduzco Flores, Jun Tani,
• Abstract summary: We propose a brain-inspired neural network model that integrates vision, proprioception, and language into a framework of predictive coding and active inference. Our results show that generalization in learning to unlearned verb-noun compositions, is significantly enhanced when training variations of task composition are increased.
• Score: 1.7624347338410742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans excel at applying learned behavior to unlearned situations. A crucial component of this generalization behavior is our ability to compose/decompose a whole into reusable parts, an attribute known as compositionality. One of the fundamental questions in robotics concerns this characteristic. "How can linguistic compositionality be developed concomitantly with sensorimotor skills through associative learning, particularly when individuals only learn partial linguistic compositions and their corresponding sensorimotor patterns?" To address this question, we propose a brain-inspired neural network model that integrates vision, proprioception, and language into a framework of predictive coding and active inference, based on the free-energy principle. The effectiveness and capabilities of this model were assessed through various simulation experiments conducted with a robot arm. Our results show that generalization in learning to unlearned verb-noun compositions, is significantly enhanced when training variations of task composition are increased. We attribute this to self-organized compositional structures in linguistic latent state space being influenced significantly by sensorimotor learning. Ablation studies show that visual attention and working memory are essential to accurately generate visuo-motor sequences to achieve linguistically represented goals. These insights advance our understanding of mechanisms underlying development of compositionality through interactions of linguistic and sensorimotor experience.

Related papers

• Constructive Approach to Bidirectional Causation between Qualia Structure and Language Emergence [5.906966694759679]
  This paper presents a novel perspective on the bidirectional causation between language emergence and relational structure of subjective experiences. We hypothesize that languages with distributional semantics, e.g., syntactic-semantic structures, may have emerged through the process of aligning internal representations among individuals.
  arXiv  Detail & Related papers  (2024-09-14T11:03:12Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• On the Correspondence between Compositionality and Imitation in Emergent Neural Communication [1.4610038284393165]
  Our work explores the link between compositionality and imitation in a Lewis game played by deep neural agents. supervised learning tends to produce more average languages, while reinforcement learning introduces a selection pressure toward more compositional languages.
  arXiv  Detail & Related papers  (2023-05-22T11:41:29Z)
• Interpreting Neural Policies with Disentangled Tree Representations [58.769048492254555]
  We study interpretability of compact neural policies through the lens of disentangled representation. We leverage decision trees to obtain factors of variation for disentanglement in robot learning. We introduce interpretability metrics that measure disentanglement of learned neural dynamics.
  arXiv  Detail & Related papers  (2022-10-13T01:10:41Z)
• Data-driven emotional body language generation for social robotics [58.88028813371423]
  In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration. We implement a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions. The evaluation study found that the anthropomorphism and animacy of the generated expressions are not perceived differently from the hand-designed ones.
  arXiv  Detail & Related papers  (2022-05-02T09:21:39Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Compositional Processing Emerges in Neural Networks Solving Math Problems [100.80518350845668]
  Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed. Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
  arXiv  Detail & Related papers  (2021-05-19T07:24:42Z)
• Crossmodal Language Grounding in an Embodied Neurocognitive Model [28.461246169379685]
  Human infants are able to acquire natural language seemingly easily at an early age. From a neuroscientific perspective, natural language is embodied, grounded in most, if not all, sensory and sensorimotor modalities. We present a neurocognitive model for language grounding which reflects bio-inspired mechanisms.
  arXiv  Detail & Related papers  (2020-06-24T08:12:09Z)
• Towards a self-organizing pre-symbolic neural model representing sensorimotor primitives [15.364871660385155]
  The acquisition of symbolic and linguistic representations of sensorimotor behavior is a cognitive process performed by an agent. We propose a model that relates the conceptualization of the higher-level information from visual stimuli to the development of ventral/dorsal visual streams. We exemplify this model through a robot passively observing an object to learn its features and movements.
  arXiv  Detail & Related papers  (2020-06-20T01:58:28Z)
• Compositional Generalization by Learning Analytical Expressions [87.15737632096378]
  A memory-augmented neural model is connected with analytical expressions to achieve compositional generalization. Experiments on the well-known benchmark SCAN demonstrate that our model seizes a great ability of compositional generalization.
  arXiv  Detail & Related papers  (2020-06-18T15:50:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.