How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-perspective Evaluation Framework

• URL: http://arxiv.org/abs/2608.01592v1
• Date: Mon, 03 Aug 2026 01:56:08 GMT
• Title: How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-perspective Evaluation Framework
• Abstract summary: Taxonomies provide a shared conceptual framework for organizing heterogeneous observations in software engineering (SE) research.<n>While advances in Large Language Models (LLMs) have led to the emergence of automated taxonomy generation methods outside the SE domain, their applicability to technically complex SE artifacts remains unclear.<n>We present the first comprehensive empirical evaluation of how state-of-the-art automated methods perform on SE artifacts.
• Score: 4.372207458643732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Taxonomies provide a shared conceptual framework for organizing heterogeneous observations in software engineering (SE) research. Manually constructing such taxonomies is labor-intensive and requires annotators with expertise in the SE domain. While advances in Large Language Models (LLMs) have led to the emergence of automated taxonomy generation methods outside the SE domain, their applicability to technically complex SE artifacts remains unclear. In this experience paper, we present the first comprehensive empirical evaluation of how state-of-the-art automated methods perform on SE artifacts through a multi-perspective evaluation framework, including taxonomy quality, alignment with taxonomies defined by human experts, reliability under independent annotation, and efficiency. To support this evaluation, we systematically collect seven SE papers with publicly available artifacts and human-defined taxonomies, and conduct experiments using two automated methods (TnT-LLM and CLIMB) with five state-of-the-art LLMs. Our evaluation reveals a clear trade-off: TnT-LLM constructs high-quality taxonomies comparable to human-defined ones but incurs substantially higher cost and runtime and tends to generate overly complex taxonomies, whereas CLIMB is 15--40$\times$ faster and 8--49$\times$ cheaper but tends to score lower on quality when technical inference beyond surface-level similarity is required. These findings suggest that TnT-LLM and CLIMB can be used in practical situations in the SE domain, while researchers should first assess the complexity of the generated taxonomies and their cost using a subset of the target data to decide whether to use automated methods or human experts. Our work represents a first step toward a systematic understanding of automated taxonomy generation in SE, offering actionable insights for future research and practice.

Related papers

• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• Towards a Science of Collective AI: LLM-based Multi-Agent Systems Need a Transition from Blind Trial-and-Error to Rigorous Science [70.3658845234978]
  Large Language Models (LLMs) have greatly extended the capabilities of Multi-Agent Systems (MAS)<n>Despite this rapid progress, the field still relies heavily on empirical trial-and-error.<n>This bottleneck stems from the ambiguity of attribution.<n>We propose a factor attribution paradigm to systematically identify collaboration-driving factors.
  arXiv  Detail & Related papers  (2026-02-05T04:19:52Z)
• TaxoAlign: Scholarly Taxonomy Generation Using Language Models [14.54227999705238]
  Taxonomies play a crucial role in helping researchers structure and navigate knowledge in a hierarchical manner.<n>Existing approaches to automatic survey generation do not compare the structure of the generated surveys with those written by human experts.<n>We present our own method for automated taxonomy creation that can bridge the gap between human-generated and automatically-created.
  arXiv  Detail & Related papers  (2025-10-20T07:49:51Z)
• Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering [59.54662810933882]
  Existing taxonomy construction methods, leveraging unsupervised clustering or direct prompting of large language models, often lack coherence and granularity.<n>We propose a novel context-aware hierarchical taxonomy generation framework that integrates LLM-guided multi-aspect encoding with dynamic clustering.
  arXiv  Detail & Related papers  (2025-09-23T15:12:58Z)
• Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration [10.386888517619997]
  We introduce CLIMB, a framework that automates the creation of high-quality, data-driven occupations.<n>On three diverse, real-world datasets, we show that CLIMB produces that are more coherent and scalable than existing methods.
  arXiv  Detail & Related papers  (2025-09-19T09:17:48Z)
• Transforming Expert Knowledge into Scalable Ontology via Large Language Models [0.0]
  Traditional approaches to taxonomy alignment rely on expert review of concept pairs.<n>We propose a novel framework that combines large language models (LLMs) with expert calibration and iterative prompt optimization.<n>In evaluating our framework on a domain-specific mapping task of concept essentiality, we achieved an F1-score of 0.97, substantially exceeding the human benchmark of 0.68.
  arXiv  Detail & Related papers  (2025-06-10T03:48:26Z)
• An AI-Powered Research Assistant in the Lab: A Practical Guide for Text Analysis Through Iterative Collaboration with LLMs [0.7255608805275865]
  Here we present a step-by-step tutorial to efficiently develop, test, and apply for analyzing unstructured data using LLMs.<n>We demonstrate how to write prompts to review datasets and generate a taxonomy of life domains, evaluate and refine the taxonomy through prompt and direct modifications, test the taxonomy and assess intercoder agreements, and apply the taxonomy to categorize an entire dataset with high intercoder reliability.
  arXiv  Detail & Related papers  (2025-05-14T18:32:18Z)
• SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
  textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
  arXiv  Detail & Related papers  (2025-05-10T16:52:40Z)
• Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmark [63.97125827026949]
  This paper explores the feasibility of using text-to-image models in a zero-shot setup to generate images for taxonomy concepts.<n>A benchmark is proposed that assesses models' abilities to understand taxonomy concepts and generate relevant, high-quality images.<n>The 12 models are evaluated using 9 novel taxonomy-related text-to-image metrics and human feedback.
  arXiv  Detail & Related papers  (2025-03-13T13:37:54Z)
• Prompting or Fine-tuning? A Comparative Study of Large Language Models for Taxonomy Construction [0.8670827427401335]
  We present a general framework for taxonomy construction that takes into account structural constraints. We compare the prompting and fine-tuning approaches performed on a hypernym taxonomy and a novel computer science taxonomy dataset.
  arXiv  Detail & Related papers  (2023-09-04T16:53:17Z)
• Octet: Online Catalog Taxonomy Enrichment with Self-Supervision [67.26804972901952]
  We present a self-supervised end-to-end framework, Octet for Online Catalog EnrichmenT. We propose to train a sequence labeling model for term extraction and employ graph neural networks (GNNs) to capture the taxonomy structure. Octet enriches an online catalog in production to 2 times larger in the open-world evaluation.
  arXiv  Detail & Related papers  (2020-06-18T04:53:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.