"What's my model inside of?": Exploring the role of environments for grounded natural language understanding

• URL: http://arxiv.org/abs/2402.02548v1
• Date: Sun, 4 Feb 2024 15:52:46 GMT
• Title: "What's my model inside of?": Exploring the role of environments for grounded natural language understanding
• Authors: Ronen Tamari
• Abstract summary: In this thesis we adopt an ecological approach to grounded natural language understanding (NLU) research. We develop novel training and annotation approaches for procedural text understanding based on text-based game environments. We propose a design for AI-augmented "social thinking environments" for knowledge workers like scientists.
• Score: 1.8829370712240063
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In contrast to classical cognitive science which studied brains in isolation, ecological approaches focused on the role of the body and environment in shaping cognition. Similarly, in this thesis we adopt an ecological approach to grounded natural language understanding (NLU) research. Grounded language understanding studies language understanding systems situated in the context of events, actions and precepts in naturalistic/simulated virtual environments. Where classic research tends to focus on designing new models and optimization methods while treating environments as given, we explore the potential of environment design for improving data collection and model development. We developed novel training and annotation approaches for procedural text understanding based on text-based game environments. We also drew upon embodied cognitive linguistics literature to propose a roadmap for grounded NLP research, and to inform the development of a new benchmark for measuring the progress of large language models on challenging commonsense reasoning tasks. We leveraged the richer supervision provided by text-based game environments to develop Breakpoint Transformers, a novel approach to modeling intermediate semantic information in long narrative or procedural texts. Finally, we integrated theories on the role of environments in collective human intelligence to propose a design for AI-augmented "social thinking environments" for knowledge workers like scientists.

Related papers

• How language models extrapolate outside the training data: A case study in Textualized Gridworld [32.5268320198854]
  We show that conventional approaches, including next-token prediction and Chain of Thought fine-tuning, fail to generalize in larger, unseen environments. Inspired by human cognition and dual-process theory, we propose language models should construct cognitive maps before interaction.
  arXiv  Detail & Related papers  (2024-06-21T16:10:05Z)
• Language Evolution with Deep Learning [49.879239655532324]
  Computational modeling plays an essential role in the study of language emergence. It aims to simulate the conditions and learning processes that could trigger the emergence of a structured language. This chapter explores another class of computational models that have recently revolutionized the field of machine learning: deep learning models.
  arXiv  Detail & Related papers  (2024-03-18T16:52:54Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• Embodied Agents for Efficient Exploration and Smart Scene Description [47.82947878753809]
  We tackle a setting for visual navigation in which an autonomous agent needs to explore and map an unseen indoor environment. We propose and evaluate an approach that combines recent advances in visual robotic exploration and image captioning on images. Our approach can generate smart scene descriptions that maximize semantic knowledge of the environment and avoid repetitions.
  arXiv  Detail & Related papers  (2023-01-17T19:28:01Z)
• Improving Policy Learning via Language Dynamics Distillation [87.27583619910338]
  We propose Language Dynamics Distillation (LDD), which pretrains a model to predict environment dynamics given demonstrations with language descriptions. We show that language descriptions in demonstrations improve sample-efficiency and generalization across environments.
  arXiv  Detail & Related papers  (2022-09-30T19:56:04Z)
• A Systematic Survey of Text Worlds as Embodied Natural Language Environments [0.76146285961466]
  Text Worlds are virtual environments for embodied agents that, unlike 2D or 3D environments, are rendered exclusively using textual descriptions. These environments offer an alternative to higher-fidelity 3D environments due to their low barrier to entry.
  arXiv  Detail & Related papers  (2021-07-08T22:15:16Z)
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey [54.34370423151014]
  This paper surveys the components of modeling approaches relaying task impacts across various generation tasks such as storytelling, summarization, translation etc. We present an abstraction of the imperative techniques with respect to learning paradigms, pretraining, modeling approaches, decoding and the key challenges outstanding in the field in each of them.
  arXiv  Detail & Related papers  (2020-10-14T17:54:42Z)
• Zero-Shot Compositional Policy Learning via Language Grounding [13.45138913186308]
  Humans can adapt to new tasks quickly by leveraging prior knowledge about the world such as language descriptions. We introduce a new research platform BabyAI++ in which the dynamics of environments are disentangled from visual appearance. We find that current language-guided RL/IL techniques overfit to the training environments and suffer from a huge performance drop when facing unseen combinations.
  arXiv  Detail & Related papers  (2020-04-15T16:58:19Z)
• Ecological Semantics: Programming Environments for Situated Language Understanding [25.853707930426175]
  Grounded language learning approaches offer the promise of deeper understanding by situating learning in richer, more structured training environments. We propose treating environments as "first-class citizens" in semantic representations. We argue that models must begin to understand and program in the language of affordances.
  arXiv  Detail & Related papers  (2020-03-10T08:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.