ASTAGEN: Empirical Evaluation of Automated SATD Taxonomy Generation with LLMs

• URL: http://arxiv.org/abs/2506.09601v1
• Date: Wed, 11 Jun 2025 10:59:57 GMT
• Title: ASTAGEN: Empirical Evaluation of Automated SATD Taxonomy Generation with LLMs
• Authors: Sota Nakashima, Yuta Ishimoto, Masanari Kondo, Tao Xiao, Yasutaka Kamei,
• Abstract summary: Self-admitted technical debt (SATD) refers to suboptimal code that degrades software quality.<n>This study presents ASTAGEN, an initial step toward automating SATD taxonomy generation using large language models (LLMs)<n>We evaluate ASTAGEN on SATD datasets from three domains: quantum software, smart contracts, and machine learning.
• Score: 2.287480001913659
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. This study presents ASTAGEN, an initial step toward automating SATD taxonomy generation using large language models (LLMs). Given a comment and its surrounding code, ASTAGEN first generates a concise explanation for each SATD comment, then incrementally generates and updates categories to construct a taxonomy. We evaluate ASTAGEN on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovers domain-specific categories reported in prior work, such as Layer Configuration in machine learning. Compared to a naive use of an LLM, ASTAGEN produces more consistent category assignments due to its explanation-driven, iterative design. It also completes taxonomy generation in under two hours and for less than one USD, even on the largest dataset. These results suggest that while full automation remains challenging, ASTAGEN is able to support semi-automated taxonomy construction. Furthermore, our work opens up avenues for future work, such as automatic taxonomy generation in other areas.

Related papers

• Exploring a Large Language Model for Transforming Taxonomic Data into OWL: Lessons Learned and Implications for Ontology Development [63.74965026095835]
  This paper investigates the use of ChatGPT-4 to automate the development of the :Organism module in the Agricultural Product Types Ontology (APTO) for species classification.<n>Our methodology involved leveraging ChatGPT-4 to extract data from the GBIF Backbone API and generate files for further integration in APTO.<n>Two alternative approaches were explored: (1) issuing a series of prompts for ChatGPT-4 to execute tasks via the BrowserOP plugin and (2) directing ChatGPT-4 to design a Python algorithm to perform tasks.
  arXiv  Detail & Related papers  (2025-04-25T19:05:52Z)
• Refining Wikidata Taxonomy using Large Language Models [2.392329079182226]
  We present WiKC, a new version of Wikidata taxonomy cleaned automatically using a combination of Large Language Models (LLMs) and graph mining techniques. Operations on the taxonomy, such as cutting links or merging classes, are performed with the help of zero-shot prompting on an open-source LLM.
  arXiv  Detail & Related papers  (2024-09-06T06:53:45Z)
• AutoBencher: Towards Declarative Benchmark Construction [74.54640925146289]
  We use AutoBencher to create datasets for math, multilinguality, knowledge, and safety.<n>The scalability of AutoBencher allows it to test fine-grained categories knowledge, creating datasets that elicit 22% more model errors (i.e., difficulty) than existing benchmarks.
  arXiv  Detail & Related papers  (2024-07-11T10:03:47Z)
• Using Zero-shot Prompting in the Automatic Creation and Expansion of Topic Taxonomies for Tagging Retail Banking Transactions [0.0]
  This work presents an unsupervised method for constructing and expanding topic using instruction-based fine-tuned LLMs (Large Language Models) To expand an existing taxonomy with new terms, we use zero-shot prompting to find out where to add new nodes. We use the resulting tags to assign tags that characterize merchants from a retail bank dataset.
  arXiv  Detail & Related papers  (2024-01-08T00:27:16Z)
• DAnTE: a taxonomy for the automation degree of software engineering tasks [2.356908851188234]
  We propose DAnTE, a Degree of Automation taxonomy for software engineering. We evaluate several tools used in the past and in the present for software engineering practices. We discuss what novel tools could emerge in the middle and long term.
  arXiv  Detail & Related papers  (2023-09-26T13:04:58Z)
• Towards Automatically Addressing Self-Admitted Technical Debt: How Far Are We? [17.128428286986573]
  This paper empirically investigates the extent to which technical debt can be automatically paid back by neural-based generative models. We start by extracting a dateset of 5,039 Self-Admitted Technical Debt (SATD) removals from 595 open-source projects. We use this dataset to experiment with seven different generative deep learning (DL) model configurations.
  arXiv  Detail & Related papers  (2023-08-17T12:27:32Z)
• RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap [0.0]
  We argue that automatic taxonomy evaluation (ATE) is just as important as taxonomy construction. We propose RaTE, an automatic label-free taxonomy scoring procedure, which relies on a large pre-trained language model.
  arXiv  Detail & Related papers  (2023-07-19T01:37:31Z)
• BigIssue: A Realistic Bug Localization Benchmark [89.8240118116093]
  BigIssue is a benchmark for realistic bug localization. We provide a general benchmark with a diversity of real and synthetic Java bugs. We hope to advance the state of the art in bug localization, in turn improving APR performance and increasing its applicability to the modern development cycle.
  arXiv  Detail & Related papers  (2022-07-21T20:17:53Z)
• Who Should Go First? A Self-Supervised Concept Sorting Model for Improving Taxonomy Expansion [50.794640012673064]
  As data and business scope grow in real applications, existing need to be expanded to incorporate new concepts. Previous works on taxonomy expansion process the new concepts independently and simultaneously, ignoring the potential relationships among them and the appropriate order of inserting operations. We propose TaxoOrder, a novel self-supervised framework that simultaneously discovers the local hypernym-hyponym structure among new concepts and decides the order of insertion.
  arXiv  Detail & Related papers  (2021-04-08T11:00:43Z)
• Octet: Online Catalog Taxonomy Enrichment with Self-Supervision [67.26804972901952]
  We present a self-supervised end-to-end framework, Octet for Online Catalog EnrichmenT. We propose to train a sequence labeling model for term extraction and employ graph neural networks (GNNs) to capture the taxonomy structure. Octet enriches an online catalog in production to 2 times larger in the open-world evaluation.
  arXiv  Detail & Related papers  (2020-06-18T04:53:07Z)
• TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network [62.12557274257303]
  Taxonomies consist of machine-interpretable semantics and provide valuable knowledge for many web applications. We propose a novel self-supervised framework, named TaxoExpan, which automatically generates a set of query concept, anchor concept> pairs from the existing taxonomy as training data. We develop two innovative techniques in TaxoExpan: (1) a position-enhanced graph neural network that encodes the local structure of an anchor concept in the existing taxonomy, and (2) a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data.
  arXiv  Detail & Related papers  (2020-01-26T21:30:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.