AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph

• URL: http://arxiv.org/abs/2311.09174v3
• Date: Mon, 1 Apr 2024 16:24:24 GMT
• Title: AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph
• Authors: Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, Xin Liu, Yangqiu Song,
• Abstract summary: abstraction ability is essential in human intelligence, which remains under-explored in language models. We present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge.
• Score: 62.685920585838616
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Cognitive research indicates that abstraction ability is essential in human intelligence, which remains under-explored in language models. In this paper, we present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge. While existing resources only touch nouns or verbs within simplified events or specific domains, AbsPyramid collects abstract knowledge for three components of diverse events to comprehensively evaluate the abstraction ability of language models in the open domain. Experimental results demonstrate that current LLMs face challenges comprehending abstraction knowledge in zero-shot and few-shot settings. By training on our rich abstraction knowledge, we find LLMs can acquire basic abstraction abilities and generalize to unseen events. In the meantime, we empirically show that our benchmark is comprehensive to enhance LLMs across two previous abstraction tasks.

Related papers

• Learning with Language-Guided State Abstractions [58.199148890064826]
  Generalizable policy learning in high-dimensional observation spaces is facilitated by well-designed state representations. Our method, LGA, uses a combination of natural language supervision and background knowledge from language models to automatically build state representations tailored to unseen tasks. Experiments on simulated robotic tasks show that LGA yields state abstractions similar to those designed by humans, but in a fraction of the time.
  arXiv  Detail & Related papers  (2024-02-28T23:57:04Z)
• AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation [60.40409210088717]
  Abstraction ability is crucial in human intelligence, which can also benefit various tasks in NLP study. Existing work shows that LLMs are deficient in abstract ability, and how to improve it remains unexplored. We design the framework AbsInstruct to enhance LLMs' abstraction ability through instruction tuning.
  arXiv  Detail & Related papers  (2024-02-16T12:47:11Z)
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning [63.58935783293342]
  Causal Bisimulation Modeling (CBM) is a method that learns the causal relationships in the dynamics and reward functions for each task to derive a minimal, task-specific abstraction. CBM's learned implicit dynamics models identify the underlying causal relationships and state abstractions more accurately than explicit ones.
  arXiv  Detail & Related papers  (2024-01-23T05:43:15Z)
• Neural Causal Abstractions [63.21695740637627]
  We develop a new family of causal abstractions by clustering variables and their domains. We show that such abstractions are learnable in practical settings through Neural Causal Models. Our experiments support the theory and illustrate how to scale causal inferences to high-dimensional settings involving image data.
  arXiv  Detail & Related papers  (2024-01-05T02:00:27Z)
• Emergence and Function of Abstract Representations in Self-Supervised Transformers [0.0]
  We study the inner workings of small-scale transformers trained to reconstruct partially masked visual scenes. We show that the network develops intermediate abstract representations, or abstractions, that encode all semantic features of the dataset. Using precise manipulation experiments, we demonstrate that abstractions are central to the network's decision-making process.
  arXiv  Detail & Related papers  (2023-12-08T20:47:15Z)
• Does Deep Learning Learn to Abstract? A Systematic Probing Framework [69.2366890742283]
  Abstraction is a desirable capability for deep learning models, which means to induce abstract concepts from concrete instances and flexibly apply them beyond the learning context. We introduce a systematic probing framework to explore the abstraction capability of deep learning models from a transferability perspective.
  arXiv  Detail & Related papers  (2023-02-23T12:50:02Z)
• A learning perspective on the emergence of abstractions: the curious case of phonemes [2.580765958706854]
  We test two opposing principles regarding the development of language knowledge in linguistically untrained language users. We probed whether MBL and ECL could give rise to a type of language knowledge that resembles linguistic abstractions. We show that ECL learning models can learn abstractions and that at least part of the phone inventory can be reliably identified from the input.
  arXiv  Detail & Related papers  (2020-12-14T13:33:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.