Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning

• URL: http://arxiv.org/abs/2502.14086v1
• Date: Wed, 19 Feb 2025 20:20:24 GMT
• Title: Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning
• Authors: Cole Gawin, Yidan Sun, Mayank Kejriwal,
• Abstract summary: Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving problems of moderate complexity. We systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph.
• Score: 5.4141465747474475
• License:
• Abstract: Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving reasoning tasks of moderate complexity, such as question-answering and mathematical problem-solving. However, their capabilities in tasks requiring deeper cognitive skills, such as common-sense understanding and abstract reasoning, remain under-explored. In this paper, we systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph. We propose two prompting approaches: instruct prompting, where models predict plausible semantic relationships based on provided definitions, and few-shot prompting, where models identify relations using examples as guidance. Our experiments with the gpt-4o-mini model show that in instruct prompting, consistent performance is obtained when ranking multiple relations but with substantial decline when the model is restricted to predicting only one relation. In few-shot prompting, the model's accuracy improves significantly when selecting from five relations rather than the full set, although with notable bias toward certain relations. These results suggest significant gaps still, even in commercially used LLMs' abstract common-sense reasoning abilities, compared to human-level understanding. However, the findings also highlight the promise of careful prompt engineering, based on selective retrieval, for obtaining better performance.

Related papers

• Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models [62.12031550252253]
  We present Path-of-Thoughts (PoT), a novel framework designed to tackle relation reasoning. PoT efficiently extracts a task-agnostic graph that identifies crucial entities, relations, and attributes within the problem context. PoT identifies relevant reasoning chains within the graph corresponding to the posed question, facilitating inference of potential answers.
  arXiv  Detail & Related papers  (2024-12-23T20:27:12Z)
• KAAE: Numerical Reasoning for Knowledge Graphs via Knowledge-aware Attributes Learning [26.605386507466733]
  Numerical reasoning is pivotal in various artificial intelligence applications, such as natural language processing and recommender systems. Existing approaches encounter two critical challenges in modeling: semantic relevance and semantic ambiguity. We propose the novel Knowledge-Aware Attributes Embedding model (KAAE) for knowledge graph embeddings in numerical reasoning.
  arXiv  Detail & Related papers  (2024-11-20T00:47:03Z)
• Verbalized Probabilistic Graphical Modeling with Large Language Models [8.961720262676195]
  This work introduces a novel Bayesian prompting approach that facilitates training-free Bayesian inference with large language models. Our results indicate that the model effectively enhances confidence elicitation and text generation quality, demonstrating its potential to improve AI language understanding systems.
  arXiv  Detail & Related papers  (2024-06-08T16:35:31Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction [33.528688487954454]
  Relation extraction (RE) aims to identify semantic relationships between entities within text. Few-shot learning, aiming to lessen annotation demands, typically provides incomplete and biased supervision for target relations. We introduce REPaL, comprising three stages: (1) We leverage large language models (LLMs) to generate initial seed instances from relation definitions and an unlabeled corpus.
  arXiv  Detail & Related papers  (2024-02-17T00:20:06Z)
• Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition [4.7938839332508945]
  We propose a Prompt-based Logical Semantics Enhancement (PLSE) method for Implicit Discourse Relation Recognition (IDRR) Our method seamlessly injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction. Experimental results on PDTB 2.0 and CoNLL16 datasets demonstrate that our method achieves outstanding and consistent performance against the current state-of-the-art models.
  arXiv  Detail & Related papers  (2023-11-01T08:38:08Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction [84.64435075778988]
  We propose a general approach to learn relation prototypes from unlabeled texts. We learn relation prototypes as an implicit factor between entities. We conduct experiments on two publicly available datasets: New York Times and Google Distant Supervision.
  arXiv  Detail & Related papers  (2020-11-27T06:21:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.