Passive learning of active causal strategies in agents and language models

• URL: http://arxiv.org/abs/2305.16183v2
• Date: Mon, 2 Oct 2023 13:04:59 GMT
• Title: Passive learning of active causal strategies in agents and language models
• Authors: Andrew Kyle Lampinen and Stephanie C Y Chan and Ishita Dasgupta and Andrew J Nam and Jane X Wang
• Abstract summary: We show that purely passive learning can in fact allow an agent to learn generalizable strategies for determining and using causal structures. We show that agents trained via imitation on expert data can indeed generalize at test time to infer and use causal links which are never present in the training data. Explanations can even allow passive learners to generalize out-of-distribution from perfectly-confounded training data.
• Score: 15.086300301260811
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: What can be learned about causality and experimentation from passive data? This question is salient given recent successes of passively-trained language models in interactive domains such as tool use. Passive learning is inherently limited. However, we show that purely passive learning can in fact allow an agent to learn generalizable strategies for determining and using causal structures, as long as the agent can intervene at test time. We formally illustrate that learning a strategy of first experimenting, then seeking goals, can allow generalization from passive learning in principle. We then show empirically that agents trained via imitation on expert data can indeed generalize at test time to infer and use causal links which are never present in the training data; these agents can also generalize experimentation strategies to novel variable sets never observed in training. We then show that strategies for causal intervention and exploitation can be generalized from passive data even in a more complex environment with high-dimensional observations, with the support of natural language explanations. Explanations can even allow passive learners to generalize out-of-distribution from perfectly-confounded training data. Finally, we show that language models, trained only on passive next-word prediction, can generalize causal intervention strategies from a few-shot prompt containing examples of experimentation, together with explanations and reasoning. These results highlight the surprising power of passive learning of active causal strategies, and may help to understand the behaviors and capabilities of language models.

Related papers

• Eliciting Language Model Behaviors with Investigator Agents [93.34072434845162]
  Language models exhibit complex, diverse behaviors when prompted with free-form text. We study the problem of behavior elicitation, where the goal is to search for prompts that induce specific target behaviors. We train investigator models to map randomly-chosen target behaviors to a diverse distribution of outputs that elicit them.
  arXiv  Detail & Related papers  (2025-02-03T10:52:44Z)
• Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation [7.647395374489533]
  We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions. We show that our approach generates explanations as helpful as those produced by a human domain expert.
  arXiv  Detail & Related papers  (2023-11-29T20:16:23Z)
• Reinforcement Learning Fine-tuning of Language Models is Biased Towards More Extractable Features [0.5937476291232802]
  We investigate whether principles governing inductive biases in the supervised fine-tuning of large language models also apply when the fine-tuning process uses reinforcement learning. We find statistically significant correlations which constitute strong evidence for these hypotheses.
  arXiv  Detail & Related papers  (2023-11-07T15:00:39Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Explaining Agent Behavior with Large Language Models [7.128139268426959]
  We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions. We show how a compact representation of the agent's behavior can be learned and used to produce plausible explanations.
  arXiv  Detail & Related papers  (2023-09-19T06:13:24Z)
• Reinforcement Learning from Passive Data via Latent Intentions [86.4969514480008]
  We show that passive data can still be used to learn features that accelerate downstream RL. Our approach learns from passive data by modeling intentions. Our experiments demonstrate the ability to learn from many forms of passive data, including cross-embodiment video data and YouTube videos.
  arXiv  Detail & Related papers  (2023-04-10T17:59:05Z)
• Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering [71.15403434929915]
  We show that across 5 models and 4 datasets on the task of visual question answering, a wide variety of active learning approaches fail to outperform random selection. We identify the problem as collective outliers -- groups of examples that active learning methods prefer to acquire but models fail to learn. We show that active learning sample efficiency increases significantly as the number of collective outliers in the active learning pool decreases.
  arXiv  Detail & Related papers  (2021-07-06T00:52:11Z)
• Data-driven Analysis for Understanding Team Sports Behaviors [1.1844977816228044]
  Rules regarding the real-world biological multi-agent behaviors such as team sports are often largely unknown. Estimation of the rules from data, i.e., data-driven approaches such as machine learning, provides an effective way for the analysis of such behaviors. This survey focuses on data-driven analysis for quantitative understanding of invasion team sports behaviors such as basketball and football.
  arXiv  Detail & Related papers  (2021-02-15T13:31:45Z)
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering [80.60605604261416]
  We propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. We vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. We show that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks.
  arXiv  Detail & Related papers  (2020-11-07T22:52:21Z)
• Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge [96.92252296244233]
  Large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. We show that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
  arXiv  Detail & Related papers  (2020-06-11T17:02:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.