Evidence of Meaning in Language Models Trained on Programs
- URL: http://arxiv.org/abs/2305.11169v2
- Date: Wed, 24 May 2023 11:52:13 GMT
- Title: Evidence of Meaning in Language Models Trained on Programs
- Authors: Charles Jin, Martin Rinard
- Abstract summary: We present evidence that language models can learn meaning despite being trained only to perform next token prediction on text.
We first train a Transformer model on the corpus of programs, then probe the trained model's hidden states as it completes a program given a specification.
There is a strong, statistically significant correlation between the accuracy of the probe and the model's ability to generate a program that implements the specification.
- Score: 5.892876463573452
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We present evidence that language models can learn meaning despite being
trained only to perform next token prediction on text, specifically a corpus of
programs. Each program is preceded by a specification in the form of (textual)
input-output examples. Working with programs enables us to precisely define
concepts relevant to meaning in language (e.g., correctness and semantics),
making program synthesis well-suited as an intermediate testbed for
characterizing the presence (or absence) of meaning in language models.
We first train a Transformer model on the corpus of programs, then probe the
trained model's hidden states as it completes a program given a specification.
Despite providing no inductive bias toward learning the semantics of the
language, we find that a linear probe is able to extract abstractions of both
current and future program states from the model states. Moreover, there is a
strong, statistically significant correlation between the accuracy of the probe
and the model's ability to generate a program that implements the
specification. To evaluate whether the semantics are represented in the model
states rather than learned by the probe, we design a novel experimental
procedure that intervenes on the semantics of the language while preserving the
lexicon and syntax. We also demonstrate that the model learns to generate
correct programs that are, on average, shorter than those in the training set,
which is evidence that language model outputs may differ from the training
distribution in semantically meaningful ways. In summary, this paper does not
propose any new techniques for training language models, but develops an
experimental framework for and provides insights into the acquisition and
representation of (formal) meaning in language models.
Related papers
- Learning Phonotactics from Linguistic Informants [54.086544221761486]
Our model iteratively selects or synthesizes a data-point according to one of a range of information-theoretic policies.
We find that the information-theoretic policies that our model uses to select items to query the informant achieve sample efficiency comparable to, or greater than, fully supervised approaches.
arXiv Detail & Related papers (2024-05-08T00:18:56Z) - Towards Understanding What Code Language Models Learned [10.989953856458996]
Pre-trained language models are effective in a variety of natural language tasks.
It has been argued their capabilities fall short of fully learning meaning or understanding language.
We investigate their ability to capture semantics of code beyond superficial frequency and co-occurrence.
arXiv Detail & Related papers (2023-06-20T23:42:14Z) - LaMPP: Language Models as Probabilistic Priors for Perception and Action [38.07277869107474]
We show how to leverage language models for non-linguistic perception and control tasks.
Our approach casts labeling and decision-making as inference in probabilistic graphical models.
arXiv Detail & Related papers (2023-02-03T15:14:04Z) - Benchmarking Language Models for Code Syntax Understanding [79.11525961219591]
Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding.
In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs.
Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures.
arXiv Detail & Related papers (2022-10-26T04:47:18Z) - Toward Trustworthy Neural Program Synthesis [6.3557174349423455]
We develop an approach to estimate the probability that a program sampled from a large language model is correct.
Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave.
arXiv Detail & Related papers (2022-09-29T20:32:07Z) - Probing via Prompting [71.7904179689271]
This paper introduces a novel model-free approach to probing, by formulating probing as a prompting task.
We conduct experiments on five probing tasks and show that our approach is comparable or better at extracting information than diagnostic probes.
We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
arXiv Detail & Related papers (2022-07-04T22:14:40Z) - Few-shot Subgoal Planning with Language Models [58.11102061150875]
We show that language priors encoded in pre-trained language models allow us to infer fine-grained subgoal sequences.
In contrast to recent methods which make strong assumptions about subgoal supervision, our experiments show that language models can infer detailed subgoal sequences without any fine-tuning.
arXiv Detail & Related papers (2022-05-28T01:03:30Z) - An Application of Pseudo-Log-Likelihoods to Natural Language Scoring [5.382454613390483]
A language model with relatively few parameters and training steps can outperform it on a recent large data set.
We produce some absolute state-of-the-art results for common sense reasoning in binary choice tasks.
We argue that robustness of the smaller model ought to be understood in terms of compositionality.
arXiv Detail & Related papers (2022-01-23T22:00:54Z) - A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
We propose a novel latent-variable formulation for constructing intrinsic probes.
We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
arXiv Detail & Related papers (2022-01-20T15:01:12Z) - Probing Linguistic Information For Logical Inference In Pre-trained
Language Models [2.4366811507669124]
We propose a methodology for probing linguistic information for logical inference in pre-trained language model representations.
We find that (i) pre-trained language models do encode several types of linguistic information for inference, but there are also some types of information that are weakly encoded.
We have demonstrated language models' potential as semantic and background knowledge bases for supporting symbolic inference methods.
arXiv Detail & Related papers (2021-12-03T07:19:42Z) - Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods
in Natural Language Processing [78.8500633981247]
This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning"
Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly.
arXiv Detail & Related papers (2021-07-28T18:09:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.