PhysNLU: A Language Resource for Evaluating Natural Language
Understanding and Explanation Coherence in Physics
- URL: http://arxiv.org/abs/2201.04275v3
- Date: Fri, 2 Jun 2023 15:06:25 GMT
- Title: PhysNLU: A Language Resource for Evaluating Natural Language
Understanding and Explanation Coherence in Physics
- Authors: Jordan Meadows, Zili Zhou, Andre Freitas
- Abstract summary: We present a collection of datasets developed to evaluate the performance of language models in this regard.
Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse.
We present baselines that demonstrate how contemporary language models are challenged by coherence related tasks in physics.
- Score: 1.4123037008246728
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In order for language models to aid physics research, they must first encode
representations of mathematical and natural language discourse which lead to
coherent explanations, with correct ordering and relevance of statements. We
present a collection of datasets developed to evaluate the performance of
language models in this regard, which measure capabilities with respect to
sentence ordering, position, section prediction, and discourse coherence.
Analysis of the data reveals equations and sub-disciplines which are most
common in physics discourse, as well as the sentence-level frequency of
equations and expressions. We present baselines that demonstrate how
contemporary language models are challenged by coherence related tasks in
physics, even when trained on mathematical natural language objectives.
Related papers
- Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions [0.0]
Language models can hallucinate when performing complex and detailed mathematical reasoning.
We assess the ability of Language Models (LMs) to perform fine-grained mathematical and physical reasoning.
arXiv Detail & Related papers (2024-04-29T02:43:23Z) - Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language
Modelling [70.23876429382969]
We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks.
Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena.
For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
arXiv Detail & Related papers (2023-07-16T15:18:25Z) - Tree-Based Representation and Generation of Natural and Mathematical
Language [77.34726150561087]
Mathematical language in scientific communications and educational scenarios is important yet relatively understudied.
Recent works on mathematical language focus either on representing stand-alone mathematical expressions, or mathematical reasoning in pre-trained natural language models.
We propose a series of modifications to existing language models to jointly represent and generate text and math.
arXiv Detail & Related papers (2023-02-15T22:38:34Z) - Language Models as Inductive Reasoners [125.99461874008703]
We propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts.
We create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language.
We provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts.
arXiv Detail & Related papers (2022-12-21T11:12:14Z) - Overcoming Barriers to Skill Injection in Language Modeling: Case Study
in Arithmetic [14.618731441943847]
We develop a novel framework that enables language models to be mathematically proficient while retaining their linguistic prowess.
Specifically, we offer information-theoretic interventions to overcome the catastrophic forgetting of linguistic skills that occurs while injecting non-linguistic skills into language models.
arXiv Detail & Related papers (2022-11-03T18:53:30Z) - Benchmarking Language Models for Code Syntax Understanding [79.11525961219591]
Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding.
In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs.
Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures.
arXiv Detail & Related papers (2022-10-26T04:47:18Z) - A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
We propose a novel latent-variable formulation for constructing intrinsic probes.
We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
arXiv Detail & Related papers (2022-01-20T15:01:12Z) - Towards Zero-shot Language Modeling [90.80124496312274]
We construct a neural model that is inductively biased towards learning human languages.
We infer this distribution from a sample of typologically diverse training languages.
We harness additional language-specific side information as distant supervision for held-out languages.
arXiv Detail & Related papers (2021-08-06T23:49:18Z) - The Rediscovery Hypothesis: Language Models Need to Meet Linguistics [8.293055016429863]
We study whether linguistic knowledge is a necessary condition for good performance of modern language models.
We show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures.
This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objective with linguistic information.
arXiv Detail & Related papers (2021-03-02T15:57:39Z) - Modelling Compositionality and Structure Dependence in Natural Language [0.12183405753834563]
Drawing on linguistics and set theory, a formalisation of these ideas is presented in the first half of this thesis.
We see how cognitive systems that process language need to have certain functional constraints.
Using the advances of word embedding techniques, a model of relational learning is simulated.
arXiv Detail & Related papers (2020-11-22T17:28:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.