Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

• URL: http://arxiv.org/abs/2604.16258v1
• Date: Fri, 17 Apr 2026 17:16:32 GMT
• Title: Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models
• Abstract summary: This paper introduces measures for the systematic comparison of Competency Questions (CQs) across multiple dimensions.<n>We identify their salient properties, including readability, relevance with respect to the input text and structural complexity of the generated questions.
• Score: 0.11363776709575053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Competency Questions (CQs) are a cornerstone of requirement elicitation in ontology engineering. CQs represent requirements as a set of natural language questions that an ontology should satisfy; they are traditionally modelled by ontology engineers together with domain experts as part of a human-centred, manual elicitation process. The use of Generative AI automates CQ creation at scale, therefore democratising the process of generation, widening stakeholder engagement, and ultimately broadening access to ontology engineering. However, given the large and heterogeneous landscape of LLMs, varying in dimensions such as parameter scale, task and domain specialisation, and accessibility, it is crucial to characterise and understand the intrinsic, observable properties of the CQs they produce (e.g., readability, structural complexity) through a systematic, cross-domain analysis. This paper introduces a set of quantitative measures for the systematic comparison of CQs across multiple dimensions. Using CQs generated from well defined use cases and scenarios, we identify their salient properties, including readability, relevance with respect to the input text and structural complexity of the generated questions. We conduct our experiments over a set of use cases and requirements using a range of LLMs, including both open (KimiK2-1T, LLama3.1-8B, LLama3.2-3B) and closed models (Gemini 2.5 Pro, GPT 4.1). Our analysis demonstrates that LLM performance reflects distinct generation profiles shaped by the use case.

Related papers

• OntoExtend: A Framework for Requirement-driven and Scalable Ontology Extension with LLMs [4.480807656622782]
  OntoExtend is a requirements-driven framework for extension with Language Models (LLMs)<n>It uses retrieval-auged generation (RAG) over relevant input and requirements to propose grounded extensions.<n>We evaluate OntoExtend on 39 CQs from two use cases: a public EU-project, Onto-DESIDE, and an industrial ontology from Bosch.
  arXiv  Detail & Related papers  (2026-07-20T14:04:01Z)
• A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements [0.22499166814992444]
  This paper presents an empirical evaluation of three distinct CQ formulation approaches.<n>We generate CQs using each approach from a set of requirements for cultural heritage.<n>We assess them across different dimensions: degree of acceptability, ambiguity, relevance, complexity and readability.
  arXiv  Detail & Related papers  (2025-07-01T14:49:30Z)
• Evaluating Large Language Models for Real-World Engineering Tasks [75.97299249823972]
  This paper introduces a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios.<n>Using this dataset, we evaluate four state-of-the-art Large Language Models (LLMs)<n>Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.
  arXiv  Detail & Related papers  (2025-05-12T14:05:23Z)
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [93.8285345915925]
  Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making.<n>With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems.<n>We categorize existing methods along two dimensions: (1) Regimes, which define the stage at which reasoning is achieved; and (2) Architectures, which determine the components involved in the reasoning process.
  arXiv  Detail & Related papers  (2025-04-12T01:27:49Z)
• Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B [51.74607395697567]
  In-Context Learning (ICL) is an intriguing ability of large language models (LLMs)<n>We use causal interventions to identify information flow in Gemma-2 2B for five naturalistic ICL tasks.<n>We find that the model infers task information using a two-step strategy we call contextualize-then-aggregate.
  arXiv  Detail & Related papers  (2025-03-31T18:33:55Z)
• Ontology Generation using Large Language Models [1.0037949839020768]
  We present and evaluate two new prompting techniques for automated ontology development: Memoryless CQbyCQ and Ontogenia.<n>Trials show that OpenAI o1-preview with Ontogenia produces of sufficient quality to meet the requirements of engineers.
  arXiv  Detail & Related papers  (2025-03-07T13:03:28Z)
• A RAG Approach for Generating Competency Questions in Ontology Engineering [1.0044270899550196]
  With the emergence of Large Language Models (LLMs), there arises the possibility to automate and enhance this process.<n>We present a retrieval-augmented generation (RAG) approach that uses LLMs for the automatic generation of CQs.<n>We conduct experiments using GPT-4 on two domain engineering tasks and compare results against ground-truth CQs constructed by domain experts.
  arXiv  Detail & Related papers  (2024-09-13T13:34:32Z)
• IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarization [59.06663981902496]
  Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization.<n>We investigate two indispensable characteristics that the LLMs-based QFS models should be harnessed, Lengthy Document Summarization and Efficiently Fine-grained Query-LLM Alignment.<n>These innovations pave the way for broader application and accessibility in the field of QFS technology.
  arXiv  Detail & Related papers  (2024-07-15T07:14:56Z)
• An Experiment in Retrofitting Competency Questions for Existing Ontologies [0.0]
  Inspecting CQs together with the axioms provides critical insights into the scope and applicability of the CQs. CQs are integral to the majority of engineering methodologies, but the practice of publishing CQs alongside the on artefacts is not widely observed.
  arXiv  Detail & Related papers  (2023-11-09T08:57:39Z)
• An Empirical Comparison of LM-based Question and Answer Generation Methods [79.31199020420827]
  Question and answer generation (QAG) consists of generating a set of question-answer pairs given a context. In this paper, we establish baselines with three different QAG methodologies that leverage sequence-to-sequence language model (LM) fine-tuning. Experiments show that an end-to-end QAG model, which is computationally light at both training and inference times, is generally robust and outperforms other more convoluted approaches.
  arXiv  Detail & Related papers  (2023-05-26T14:59:53Z)
• Self-Prompting Large Language Models for Zero-Shot Open-Domain QA [67.08732962244301]
  Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing background documents. This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models. We propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of Large Language Models.
  arXiv  Detail & Related papers  (2022-12-16T18:23:43Z)
• ProQA: Structural Prompt-based Pre-training for Unified Question Answering [84.59636806421204]
  ProQA is a unified QA paradigm that solves various tasks through a single model. It concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios.
  arXiv  Detail & Related papers  (2022-05-09T04:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.