Comparing Human and Large Language Model Interpretation of Implicit Information

• URL: http://arxiv.org/abs/2604.17085v1
• Date: Sat, 18 Apr 2026 17:43:59 GMT
• Title: Comparing Human and Large Language Model Interpretation of Implicit Information
• Abstract summary: We introduce the task of Implicit Information Extraction (IIE)<n>IIE builds a structured knowledge graph from a context sentence by extracting relational triplets, validating implicit inferences, and analyzing temporal relations.<n>We find that humans agree with most model triplets yet consistently propose many additions, indicating limited coverage in current LLM-based IIE.
• Score: 5.232306238197686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The interpretation of implicit meanings is an integral aspect of human communication. However, this framework may not transfer to interactions with Large Language Models (LLMs). To investigate this, we introduce the task of Implicit Information Extraction (IIE) and propose an LLM-based IIE pipeline that builds a structured knowledge graph from a context sentence by extracting relational triplets, validating implicit inferences, and analyzing temporal relations. We evaluate two LLMs against crowdsourced human judgments on two datasets. We find that humans agree with most model triplets yet consistently propose many additions, indicating limited coverage in current LLM-based IIE. Moreover, in our experiments, models appear to be more conservative about implicit inferences than humans in socially rich contexts, whereas humans become more conservative in shorter, fact-oriented contexts. Our code is available at https://github.com/Antonio-Dee/IIE_from_LLM.

Related papers

• How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework [0.47954782871066737]
  We propose a context-aware evaluation framework in which human-likeness is assessed.<n>We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber.
  arXiv  Detail & Related papers  (2026-05-22T14:04:25Z)
• ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models [75.05436691700572]
  We introduce ExpliCa, a new dataset for evaluating Large Language Models (LLMs) in explicit causal reasoning.<n>We tested seven commercial and open-source LLMs on ExpliCa through prompting and perplexity-based metrics.<n>Surprisingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events.
  arXiv  Detail & Related papers  (2025-02-21T14:23:14Z)
• Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities [0.0]
  We compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants.<n>We aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities.
  arXiv  Detail & Related papers  (2025-01-22T16:07:24Z)
• WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia [59.96425443250666]
  Retrieval-augmented generation (RAG) has emerged as a promising solution to mitigate the limitations of large language models (LLMs) In this work, we conduct a comprehensive evaluation of LLM-generated answers to questions based on contradictory passages from Wikipedia. We benchmark a diverse range of both closed and open-source LLMs under different QA scenarios, including RAG with a single passage, and RAG with 2 contradictory passages.
  arXiv  Detail & Related papers  (2024-06-19T20:13:42Z)
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment [84.32768080422349]
  Alignment with human preference prevents large language models from generating misleading or toxic content. We propose a new formulation of prompt diversity, implying a linear correlation with the final performance of LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2024-03-17T07:08:55Z)
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation [94.59630161324013]
  We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
  arXiv  Detail & Related papers  (2023-10-24T08:56:49Z)
• An Investigation of LLMs' Inefficacy in Understanding Converse Relations [30.94718664430869]
  We introduce a new benchmark ConvRe focusing on converse relations, which contains 17 relations and 1240 triples extracted from knowledge graph completion datasets. Our ConvRE features two tasks, Re2Text and Text2Re, which are formulated as multi-choice question answering to evaluate LLMs' ability to determine the matching between relations and associated text.
  arXiv  Detail & Related papers  (2023-10-08T13:45:05Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs [26.118193748582197]
  We evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference, models in three of these categories perform close to random. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models.
  arXiv  Detail & Related papers  (2022-10-26T19:04:23Z)
• Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks [39.39138995087475]
  We ask how much of human-like thinking can be captured by learning statistical patterns in language alone. Our benchmark contains two problem-solving domains (planning and explanation generation) and is designed to require generalization. We find that humans are far more robust than LLMs on this benchmark.
  arXiv  Detail & Related papers  (2022-05-11T18:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.