Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI

• URL: http://arxiv.org/abs/2603.02944v1
• Date: Tue, 03 Mar 2026 12:51:54 GMT
• Title: Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI
• Abstract summary: Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts.<n>This study focuses on reducing labeling effort in ATD detection by combining keyword-based filtering with active learning and explainable AI.
• Score: 4.357716629393195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts such as code comments, commit messages, and issue trackers. Among its types, Architecture Technical Debt (ATD) is particularly difficult to detect due to its abstract and context-dependent nature. Manual annotation of ATD is costly, time-consuming, and challenging to scale. This study focuses on reducing labeling effort in ATD detection by combining keyword-based filtering with active learning and explainable AI. We refined an existing dataset of 116 ATD-related Jira issues from prior work, producing 57 expert-validated items used to extract representative keywords. These were applied to identify over 103,000 candidate issues across ten open-source projects. To assess the reliability of this keyword-based filtering, we conducted a qualitative evaluation of a statistically representative sample of labeled issues. Building on this filtered dataset, we applied active learning with multiple query strategies to prioritize the most informative samples for annotation. Our results show that the Breaking Ties strategy consistently improves model performance, achieving the highest F1-score of 0.72 while reducing the annotation effort by 49\%. In order to enhance model transparency, we applied SHAP and LIME to explain the outcomes of automated ATD classification. Expert evaluation revealed that both LIME and SHAP provided reasonable explanations, with the usefulness of the explanations often depending on the relevance of the highlighted features. Notably, experts preferred LIME overall for its clarity and ease of use.

Related papers

• Generative Explainability for Next-Generation Networks: LLM-Augmented XAI with Mutual Feature Interactions [1.139115821799478]
  Existing explainable artificial intelligence (XAI) techniques often fail to bridge this gap for non-specialists.<n>This paper presents a framework specifically designed to address this shortcoming.<n>It employs a structured prompt enriched with mutual feature interaction data to generate human-understandable natural language explanations.
  arXiv  Detail & Related papers  (2026-06-09T14:48:26Z)
• From Retrieval to Reasoning: A Framework for Cyber Threat Intelligence NER with Explicit and Adaptive Instructions [15.710492251334792]
  TTPrompt is a framework shifting from implicit induction to explicit instruction.<n> FIR enables LLMs to self-refine guidelines by learning from errors on minimal labeled data.<n>With refinement on just 1% of training data, TTPrompt rivals models fine-tuned on the full dataset.
  arXiv  Detail & Related papers  (2025-12-22T14:13:01Z)
• DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AI [24.349800949355465]
  Large language models (LLMs) often struggle to objectively identify latent characteristics in large datasets.<n>We propose Data Scientist AI (DSAI), a framework that enables unbiased and interpretable feature extraction.
  arXiv  Detail & Related papers  (2024-12-09T08:47:05Z)
• Optimizing Language Model's Reasoning Abilities with Weak Supervision [48.60598455782159]
  We present textscPuzzleBen, a weakly supervised benchmark that comprises 25,147 complex questions, answers, and human-generated rationales. A unique aspect of our dataset is the inclusion of 10,000 unannotated questions, enabling us to explore utilizing fewer supersized data to boost LLMs' inference capabilities.
  arXiv  Detail & Related papers  (2024-05-07T07:39:15Z)
• An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models [55.01592097059969]
  Supervised finetuning on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool. We propose using experimental design to circumvent the computational bottlenecks of active learning.
  arXiv  Detail & Related papers  (2024-01-12T16:56:54Z)
• Revisiting Sparse Retrieval for Few-shot Entity Linking [33.15662306409253]
  We propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression. For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions. Experimental results on the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by a significant margin across all test domains.
  arXiv  Detail & Related papers  (2023-10-19T03:51:10Z)
• Active Learning for Abstractive Text Summarization [50.79416783266641]
  We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
  arXiv  Detail & Related papers  (2023-01-09T10:33:14Z)
• Temporal Output Discrepancy for Loss Estimation-based Active Learning [65.93767110342502]
  We present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss. Our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2022-12-20T19:29:37Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Truth Discovery in Sequence Labels from Crowds [12.181422057560201]
  Crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), have been deployed to assist in this purpose. Existing literature in annotation aggregation assumes that annotations are independent and thus faces challenges when handling the sequential label aggregation tasks. We propose an optimization-based method that infers the ground truth labels using annotations provided by workers for sequential labeling tasks.
  arXiv  Detail & Related papers  (2021-09-09T19:12:13Z)
• Latent Opinions Transfer Network for Target-Oriented Opinion Words Extraction [63.70885228396077]
  We propose a novel model to transfer opinions knowledge from resource-rich review sentiment classification datasets to low-resource task TOWE. Our model achieves better performance compared to other state-of-the-art methods and significantly outperforms the base model without transferring opinions knowledge.
  arXiv  Detail & Related papers  (2020-01-07T11:50:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.