GitReq: A Gold Standard Dataset for Software Quality Requirements

• URL: http://arxiv.org/abs/2606.21810v1
• Date: Sat, 20 Jun 2026 00:05:32 GMT
• Title: GitReq: A Gold Standard Dataset for Software Quality Requirements
• Abstract summary: GitHub issue trackers contain millions of developer-written quality concerns.<n>No publicly available GitHub dataset classifies these into fine-grained software quality categories.<n>We construct and release GitReq GitHub Requirement Issue, comprising 6,302 expert-validated requirements mined from 55,588 raw GitHub candidates.
• Score: 0.28675177318965045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GitHub issue trackers contain millions of developer-written quality concerns, including performance bottlenecks and security vulnerabilities, yet no publicly available GitHub dataset classifies these into fine-grained software quality categories. We construct and release GitReq GitHub Requirement Issue, comprising 6,302 expert-validated requirements mined from 55,588 raw GitHub candidates across 4,080 repositories, labeled across eight ISO/IEC 25010:2011-aligned categories: Performance, Security, Portability, Availability, Fault-tolerance, Scalability, Maintainability, and a Functional baseline. Dataset construction involved category-specific triple-signal GitHub mining, separate non-functional requirement (NFR) and functional requirement (FR) preprocessing pipelines with per-category parameters, and expert human annotation achieving substantial inter-annotator agreement (Fleiss' Kappa~=~0.72). Zero-shot evaluation with four large language models (LLMs) establishes baselines, with GPT-5.2 reaching the highest macro-averaged F1 of 0.641. GitReq is publicly released with full materials to advance research in automated requirement classification and software quality analysis.

Related papers

• Consensus-gated Multi-Agent Neural Architecture Search for Seismic Fault Segmentation [0.17188280334580194]
  Neural networks for seismic fault segmentation are often borrowed from computer vision and medical imaging domains.<n>Classical neural architecture search (NAS) is restricted to hand-crafted search spaces and large compute budgets.<n>We present an agentic NAS system in which a panel of three large language models (Claude, GPT-5.1, and Gemini2.5Pro) debates each candidate architecture to unanimous consensus.
  arXiv  Detail & Related papers  (2026-08-14T02:41:37Z)
• An Approach for Embedding-Guided Function Reuse Detection in Embedded C Software [1.3114114427206678]
  This paper presents a domain-aware retrieval-augmented generation pipeline for embedded software reuse detection.<n>The pipeline enriches each function by extracting its existing inline comments, callgraph context, and a project before embedding it with eight backbone models.<n>The pipeline reveals that SonarQube produces a 93.6% false-positive rate as a reuse filter, with 83.5% of failures caused by hardware-environment mismatches.
  arXiv  Detail & Related papers  (2026-08-04T18:46:17Z)
• MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis [17.260324148993046]
  We introduce MindForge, an automated pipeline that converts open-source command-line programs into source-free environments.<n>We construct training environments from repositories disjoint from those in ProgramBench, and curate a high-quality data recipe.<n>Fine-tuning Qwen3.6-27B on these trajectories increases its ProgramBench average test pass rate from 37.98% to 49.51%.
  arXiv  Detail & Related papers  (2026-07-29T17:23:02Z)
• Industry Classification of GitHub Repositories Using the North American Industry Classification System (NAICS) [46.877501276559165]
  GitHub hosts hundreds of millions of public repositories, but the platform exposes no native mapping from repositories to standardized industry sectors.<n>We present NAICS-GH, a publicly released corpus of 6,588 GitHub repositories drawn from source pools covering the United States, the European Union, and Australia.<n>Labels are produced by a retrieve-and-verify pipeline that combines BAAI/bge-large-en embeddings, FAISS retrieval, and GPT-4.1 scoring.
  arXiv  Detail & Related papers  (2026-07-07T17:06:32Z)
• ATLAS: Agentic Taxonomy of Large-Scale Software Ecosystems [14.128311996090352]
  GitHub lacks a systematic hierarchical taxonomy of software repositories.<n>We present ATLAS, the first framework that automatically constructs a hierarchical taxonomy for software repositories and classifies projects into it end-to-end.
  arXiv  Detail & Related papers  (2026-06-19T16:51:10Z)
• SentTrack: Sentiment-Driven Bottleneck Detection in GitHub Issue Repositories [4.18477469430107]
  SentTrack is a framework for detecting socio-technical bottlenecks from GitHub issue discussions.<n>It is applied to the AvaloniaUI open-source repository across approximately 9,000 issue threads.
  arXiv  Detail & Related papers  (2026-06-09T22:12:41Z)
• DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch [89.87798465720181]
  We introduce textbfDeNovoSWE, a large-scale dataset for whole-repository generation.<n>DeNovoSWE comprises 4,818 high-quality instances, where each instance requires generating a complete repository from documentation.<n>Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.
  arXiv  Detail & Related papers  (2026-06-09T11:37:15Z)
• SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scale [54.70985426016736]
  We present SkillDAG, which models inter-skill relationships as a typed directed graph.<n>Each search returns vector matches, typed-edge neighbors, and conflict signals, and a propose-then-commit protocol lets the agent register execution-backed edges.<n>On ALFWorld and SkillsBench with MiniMax-M2.7, SkillDAG reaches 67.1% success and 27.3% reward.
  arXiv  Detail & Related papers  (2026-06-02T02:45:21Z)
• Weaponizing the Commons: A Taxonomy and Detection Framework of Abuse on GitHub [31.618034402766018]
  GitHub plays a critical role in modern software supply chains, making its security an important research concern.<n>Existing studies have primarily focused on CI/CD automation, collaboration patterns, and community management, while abuse behaviors on GitHub have received little systematic investigation.<n>In this paper, we systematically review and summarize reported GitHub abuse behaviors and conduct an empirical analysis of publicly available abuse cases.<n>Based on this investigation, we propose a comprehensive taxonomy that characterizes their diverse symptoms and root causes from a software security perspective.
  arXiv  Detail & Related papers  (2026-04-20T07:37:03Z)
• ReqFusion: A Multi-Provider Framework for Automated PEGS Analysis Across Software Domains [0.45880283710344066]
  ReqFusion is an AI-enhanced system that automates the extraction, classification, and analysis of software requirements.<n>The architecture integrates OpenAI GPT, Anthropic Claude, and Groq models to extract functional and non-functional requirements.
  arXiv  Detail & Related papers  (2026-03-24T17:45:40Z)
• GREPO: A Benchmark for Graph Neural Networks on Repository-Level Bug Localization [50.009407518866965]
  Repository-level bug localization is a critical software engineering challenge.<n>GNNs offer a promising alternative due to their ability to model complex, repository-wide dependencies.<n>We introduce GREPO, the first GNN benchmark for repository-scale bug localization tasks.
  arXiv  Detail & Related papers  (2026-02-14T23:22:15Z)
• RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository [52.98970048197381]
  RepoGenesis is the first multilingual benchmark for repository-level end-to-end web microservice generation.<n>It consists of 106 repositories (60 Python, 46 Java) across 18 domains and 11 frameworks, with 1,258 API endpoints and 2,335 test cases verified.<n>Results reveal that despite high AC (up to 73.91%) and DSR (up to 100%), the best-performing system achieves only 23.67% Pass@1 on Python and 21.45% on Java.
  arXiv  Detail & Related papers  (2026-01-20T13:19:20Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git [0.8397730500554048]
  GitGoodBench is a novel benchmark for evaluating AI agent performance on Version Control System (VCS) tasks.<n>Our benchmark covers three core Git scenarios extracted from open-source Python, Java, and Kotlin repositories.<n>We establish baseline performance on the prototyping version of our benchmark using GPT-4o equipped with custom tools, achieving a 21.11% solve rate overall.
  arXiv  Detail & Related papers  (2025-05-28T16:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.