SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub

• URL: http://arxiv.org/abs/2501.15922v1
• Date: Mon, 27 Jan 2025 10:17:38 GMT
• Title: SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub
• Authors: Benjamin C. Carter, Jonathan Rivas Contreras, Carlos A. Llanes Villegas, Pawan Acharya, Jack Utzerath, Adonijah O. Farner, Hunter Jenkins, Dylan Johnson, Jacob Penney, Igor Steinmacher, Marco A. Gerosa, Fabio Santos,
• Abstract summary: We introduce a novel tool, SkillScope, which retrieves current issues from Java projects hosted on GitHub and predicts the multilevel programming skills required to resolve these issues. In a case study, we demonstrate that SkillScope could predict 217 multilevel skills for tasks with 91% precision, 88% recall, and 89% F-measure on average.
• Score: 8.890715113245877
• License:
• Abstract: New contributors often struggle to find tasks that they can tackle when onboarding onto a new Open Source Software (OSS) project. One reason for this difficulty is that issue trackers lack explanations about the knowledge or skills needed to complete a given task successfully. These explanations can be complex and time-consuming to produce. Past research has partially addressed this problem by labeling issues with issue types, issue difficulty level, and issue skills. However, current approaches are limited to a small set of labels and lack in-depth details about their semantics, which may not sufficiently help contributors identify suitable issues. To surmount this limitation, this paper explores large language models (LLMs) and Random Forest (RF) to predict the multilevel skills required to solve the open issues. We introduce a novel tool, SkillScope, which retrieves current issues from Java projects hosted on GitHub and predicts the multilevel programming skills required to resolve these issues. In a case study, we demonstrate that SkillScope could predict 217 multilevel skills for tasks with 91% precision, 88% recall, and 89% F-measure on average. Practitioners can use this tool to better delegate or choose tasks to solve in OSS projects.

Related papers

• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)
• Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving [86.04158840879727]
  We develop a prompt-guided interaction procedure to get a powerful LLM to assign sensible skill labels to math questions. We then have it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans.
  arXiv  Detail & Related papers  (2024-05-20T17:45:26Z)
• Creating a Trajectory for Code Writing: Algorithmic Reasoning Tasks [0.923607423080658]
  This paper describes instruments and the machine learning models used for validating them. We have used the data collected in an introductory programming course in the penultimate week of the semester. Preliminary research suggests ART type instruments can be combined with specific machine learning models to act as an effective learning trajectory.
  arXiv  Detail & Related papers  (2024-04-03T05:07:01Z)
• Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning [64.14254712331116]
  Non-exemplar class incremental learning aims to learn both the new and old tasks without accessing any training data from the past. We propose a novel framework of fine-grained knowledge selection and restoration.
  arXiv  Detail & Related papers  (2023-12-20T02:34:11Z)
• Tag that issue: Applying API-domain labels in issue tracking systems [20.701637107734996]
  Labeling issues with the skills required to complete them can help contributors to choose tasks in Open Source Software projects. We investigate the feasibility and relevance of automatically labeling issues with what we call "API-domains," which are high-level categories of APIs. Our results show that newcomers consider API-domain labels useful in choosing tasks, (ii) labels can be predicted with a precision of 84% and a recall of 78.6% on average, (iii) the results of the predictions reached up to 71.3% in precision and 52.5% in recall when training with a project and testing in another, and (iv) project
  arXiv  Detail & Related papers  (2023-04-06T05:49:46Z)
• GiveMeLabeledIssues: An Open Source Issue Recommendation System [9.312780130838952]
  Developers often struggle to navigate an Open Source Software (OSS) project's issue-tracking system and find a suitable task. This paper presents a tool that mines project repositories and labels issues based on the skills required to solve them. GiveMeLabeledIssues facilitates matching developers' skills to tasks, reducing the burden on project maintainers.
  arXiv  Detail & Related papers  (2023-03-23T16:39:31Z)
• Supporting the Task-driven Skill Identification in Open Source Project Issue Tracking Systems [0.0]
  We investigate the automatic labeling of open issues strategy to help the contributors to pick a task to contribute. By identifying the skills, we claim the contributor candidates should pick a task more suitable. We applied quantitative studies to analyze the relevance of the labels in an experiment and compare the strategies' relative importance.
  arXiv  Detail & Related papers  (2022-11-02T14:17:22Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Can I Solve It? Identifying APIs Required to Complete OSS Task [16.13269535068818]
  We investigate the feasibility and relevance of labeling issues with the domain of the APIs required to complete the tasks. We leverage the issues' description and the project history to build prediction models, which resulted in precision up to 82% and recall up to 97.8%. Our results can inspire the creation of tools to automatically label issues, helping developers to find tasks that better match their skills.
  arXiv  Detail & Related papers  (2021-03-23T16:16:09Z)
• KILT: a Benchmark for Knowledge Intensive Language Tasks [102.33046195554886]
  We present a benchmark for knowledge-intensive language tasks (KILT) All tasks in KILT are grounded in the same snapshot of Wikipedia. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline.
  arXiv  Detail & Related papers  (2020-09-04T15:32:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.