The Accuracy, Robustness, and Readability of LLM-Generated Sustainability-Related Word Definitions

• URL: http://arxiv.org/abs/2502.00916v1
• Date: Sun, 02 Feb 2025 21:05:21 GMT
• Title: The Accuracy, Robustness, and Readability of LLM-Generated Sustainability-Related Word Definitions
• Abstract summary: A common language with standardized definitions is crucial for effective climate discussions.<n>We compared 300 official IPCC glossary definitions with those generated by GPT-4o-mini, Llama3.1 8B, and Mistral 7B, analyzing adherence, robustness, and readability using SBERT sentence embeddings.<n>The LLMs scored an average adherence of $0.57-0.59 pm 0.15$, and their definitions proved harder to read than the originals.
• Score: 0.6798775532273751
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A common language with standardized definitions is crucial for effective climate discussions. However, concerns exist about LLMs misrepresenting climate terms. We compared 300 official IPCC glossary definitions with those generated by GPT-4o-mini, Llama3.1 8B, and Mistral 7B, analyzing adherence, robustness, and readability using SBERT sentence embeddings. The LLMs scored an average adherence of $0.57-0.59 \pm 0.15$, and their definitions proved harder to read than the originals. Model-generated definitions vary mainly among words with multiple or ambiguous definitions, showing the potential to highlight terms that need standardization. The results show how LLMs could support environmental discourse while emphasizing the need to align model outputs with established terminology for clarity and consistency.

Related papers

• Large Language Models Have Unreliable Understanding of Software Engineering Terminology [5.652196062649331]
  Large Language Models (LLMs) are increasingly used in software engineering (SE)<n>There is no systematic study that determines to which degree these LLMs actually understand standardized SE terminology.<n>Lack of such understanding can lead to miscommunication and misunderstanding.
  arXiv  Detail & Related papers  (2026-07-07T08:46:49Z)
• Evaluating LLM Personalization via Semantic Constraint Verification [25.539904066738288]
  We introduce Natural Language Inference Constraint Verification (NLICV)<n>NLICV maps sentence meanings to truth-condition sets to verify personalization constraints via a Natural Language Inference (NLI) model.<n>Experiments demonstrate that NLICV aligns closely with human annotations while drastically reducing the latency and token costs associated with LLM judges.
  arXiv  Detail & Related papers  (2026-06-15T08:04:56Z)
• Bridging Natural Language and Formal Specification--Automated Translation of Software Requirements to LTL via Hierarchical Semantics Decomposition Using LLMs [10.958536923155101]
  Req2LTL is a modular framework that bridges NL and Linear Temporal Logic.<n>It achieves 88.4% semantic accuracy and 100% syntactic correctness on real-world aerospace requirements.
  arXiv  Detail & Related papers  (2025-12-19T08:25:54Z)
• Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions [46.08795043966853]
  We conduct experiments across multiple explanation benchmark datasets (general and domain-specific)<n>Our results reveal that while explicit label definitions can enhance accuracy and explainability, their integration into an LLM's task-solving processes is neither guaranteed nor consistent.<n>These findings underscore the need for a deeper understanding of how LLMs process external knowledge alongside their pre-existing capabilities.
  arXiv  Detail & Related papers  (2025-09-02T16:01:47Z)
• Simplifications are Absolutists: How Simplified Language Reduces Word Sense Awareness in LLM-Generated Definitions [2.6217304977339473]
  We investigate how simplification impacts homonym definition quality across three target groups: Normal, Simple, and ELI5.<n>Our results show that simplification drastically degrades definition completeness by neglecting polysemy, increasing the risk of misunderstanding.<n>These findings highlight the need to balance simplicity and completeness in educational NLP to ensure reliable, context-aware definitions for all learners.
  arXiv  Detail & Related papers  (2025-07-16T07:25:27Z)
• LLM-BT-Terms: Back-Translation as a Framework for Terminology Standardization and Dynamic Semantic Embedding [2.685668802278156]
  LLM-BT is a back-translation framework powered by large language models (LLMs)<n>It is designed to automate terminology verification and standardization through cross-lingual semantic alignment.<n> BLEU scores and term-level accuracy indicate strong cross-lingual robustness, with BLEU scores exceeding 0.45 and Portuguese term accuracy reaching 100 percent.
  arXiv  Detail & Related papers  (2025-06-09T19:39:09Z)
• Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments [2.3749120526936465]
  Large language models (LLMs) can explain grammatical rules, yet they often fail to apply those rules when judging sentence acceptability.<n>We present "grammar prompting", an explain-then-process paradigm.<n> Feeding an LLM's metalinguistic explanation back to the target model bridges the gap between knowing a rule and using it.
  arXiv  Detail & Related papers  (2025-06-02T22:42:33Z)
• MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs [66.14178164421794]
  We introduce MetaFaith, a novel prompt-based calibration approach inspired by human metacognition.<n>We show that MetaFaith robustly improves faithful calibration across diverse models and task domains, enabling up to 61% improvement in faithfulness.
  arXiv  Detail & Related papers  (2025-05-30T17:54:08Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system. It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• Value Compass Leaderboard: A Platform for Fundamental and Validated Evaluation of LLMs Values [76.70893269183684]
  Large Language Models (LLMs) achieve remarkable breakthroughs, aligning their values with humans has become imperative.<n>Existing evaluations focus narrowly on safety risks such as bias and toxicity.<n>Existing benchmarks are prone to data contamination.<n>The pluralistic nature of human values across individuals and cultures is largely ignored in measuring LLMs value alignment.
  arXiv  Detail & Related papers  (2025-01-13T05:53:56Z)
• Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach [7.5069214839655345]
  Large language models (LLMs) have demonstrated remarkable proficiency in machine translation (MT) We propose a multi-step prompt chain that enhances translation faithfulness by prioritizing key terms crucial for semantic accuracy. Experiments using Llama and Qwen as base models on the FLORES-200 and WMT datasets demonstrate significant improvements over baselines.
  arXiv  Detail & Related papers  (2024-11-13T05:40:24Z)
• Do LLMs Really Adapt to Domains? An Ontology Learning Perspective [2.0755366440393743]
  Large Language Models (LLMs) have demonstrated unprecedented prowess across various natural language processing tasks in various application domains. Recent studies show that LLMs can be leveraged to perform lexical semantic tasks, such as Knowledge Base Completion (KBC) or Ontology Learning (OL) This paper investigates the question: Do LLMs really adapt to domains and remain consistent in the extraction of structured knowledge, or do they only learn lexical senses instead of reasoning?
  arXiv  Detail & Related papers  (2024-07-29T13:29:43Z)
• To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity [27.10502683001428]
  This paper focuses on entity type ambiguity, analyzing the proficiency and consistency of state-of-the-art LLMs in applying factual knowledge when prompted with ambiguous entities. Experiments reveal that LLMs struggle with choosing the correct entity reading, achieving an average accuracy of only 85%, and as low as 75% with underspecified prompts.
  arXiv  Detail & Related papers  (2024-07-24T09:48:48Z)
• CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models [60.59638232596912]
  We introduce CLAMBER, a benchmark for evaluating large language models (LLMs) Building upon the taxonomy, we construct 12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries.
  arXiv  Detail & Related papers  (2024-05-20T14:34:01Z)
• Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL [78.80673954827773]
  Large Language Models (LLMs) play a crucial role in capturing structured semantics to enhance language understanding, improve interpretability, and reduce bias. We propose using Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to extract structured semantics. We find interesting potential: LLMs can indeed capture semantic structures, and scaling-up doesn't always mirror potential. We are surprised to discover that significant overlap in the errors is made by both LLMs and untrained humans, accounting for almost 30% of all errors.
  arXiv  Detail & Related papers  (2024-05-10T11:44:05Z)
• Best Practices for Text Annotation with Large Language Models [11.421942894219901]
  Large Language Models (LLMs) have ushered in a new era of text annotation. This paper proposes a comprehensive set of standards and best practices for their reliable, reproducible, and ethical use.
  arXiv  Detail & Related papers  (2024-02-05T15:43:50Z)
• Assessing the Reliability of Large Language Model Knowledge [78.38870272050106]
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? We propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs' factual reliability.
  arXiv  Detail & Related papers  (2023-10-15T12:40:30Z)
• Semantic Consistency for Assuring Reliability of Large Language Models [9.876355290198639]
  Large Language Models (LLMs) exhibit remarkable fluency and competence across various natural language tasks. We introduce a general measure of semantic consistency, and formulate multiple versions of this metric to evaluate the performance of various LLMs. We propose a novel prompting strategy, called Ask-to-Choose (A2C), to enhance semantic consistency.
  arXiv  Detail & Related papers  (2023-08-17T18:11:33Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.