Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

• URL: http://arxiv.org/abs/2606.12231v1
• Date: Wed, 10 Jun 2026 15:39:47 GMT
• Title: Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study
• Abstract summary: The adoption of AI-powered Integrated Development Environments has introduced "Rules" as a novel software artifact.<n>Despite their role in aligning AI behavior with developer intent, the taxonomy, evolution, and practical impact of these rules remain largely unexplored.
• Score: 5.478604810189007
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The adoption of AI-powered Integrated Development Environments (AI IDEs) has introduced "Rules" as a novel software artifact, allowing developers to persistently inject project-specific constraints and architectural guidelines into the context of Large Language Models (LLMs). Despite their role in aligning AI behavior with developer intent, the taxonomy, evolution, and practical impact of these rules remain largely unexplored. To bridge this gap, we conducted a mixed-methods empirical study on AI IDE rules. By mining 83 open-source projects and extracting 7,310 rules, we established a comprehensive taxonomy comprising 5 primary and 25 secondary categories. We then triangulated these artifacts with survey responses from 99 practitioners. Our analysis identified a contrast between developer priorities and actual configurations: while practitioners rate architectural constraints as highly important, rule files in repositories primarily consist of low-level workflow and code formatting constraints. Furthermore, our analysis of 1,540 rule evolution events revealed that rules are updated frequently. Repository data further indicate that rule evolution is primarily driven by constructive context expansions (29.17%) and enrichments (26.59%). In contrast, surveyed developers reported modifying rules primarily to correct AI errors (77.78%), typically by adding new negative constraints rather than editing existing ones. Finally, an artifact compliance assessment of 160 rule evolution events revealed that updating rules significantly improves the adherence of software artifacts, with the average artifact compliance rate increasing by 22.99% (from 49.14% to 72.13%) following an update. Our study provides empirical insights that can help developers optimize prompting strategies and guide tool builders in designing automated conflict-detection and context-management mechanisms for AI IDEs.

Related papers

• How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community [0.0]
  Generative AI tools such as GitHub Copilot, ChatGPT, and coding agents have rapidly become part of everyday software development.<n>This paper presents a longitudinal analysis of generative-AI-related discussions in the Visual Studio Code (VS Code) GitHub repository.
  arXiv  Detail & Related papers  (2026-09-04T03:16:08Z)
• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• Empirical Study on the Characteristics and Evolution of AI-usage in GitHub Repositories: Evidence from Code Comments [7.213400161126317]
  We analyze 35,361 GitHub code comments that explicitly reference AI use and their associated code blocks.<n>We first open-code 500 unique comments and code blocks to derive a taxonomy of AI-assisted development activities.<n>We then annotate the full dataset using two LLM-based classifiers and aggregate predictions with Dawid-Skene expectation-maximization.
  arXiv  Detail & Related papers  (2026-06-05T02:37:44Z)
• Operationalizing Ethics for AI Agents: How Developers Encode Values into Repository Context Files [11.207503404473842]
  This vision paper investigates how ethics and values are already being translated for AI agents into actionable instructions that shape agent behavior.<n>We find that developers are already embedding guidance related to fairness, accessibility, sustainability, tone, and privacy.
  arXiv  Detail & Related papers  (2026-05-07T02:03:36Z)
• The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development [0.0]
  Controlled studies report 20-56% productivity gains on well-scoped tasks, while the most rigorous RCT documents a 19% slowdown for experienced developers.<n> telemetry across 10,000+ developers shows 98% more pull requests but 91% longer review times with flat delivery metrics.<n>This paper argues these findings constitute the Productivity-Reliability Paradox (PRP): a systematic phenomenon emerging from non-deterministic code generators and insufficient specification discipline.
  arXiv  Detail & Related papers  (2026-05-01T23:37:50Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Predicting Developer Acceptance of AI-Generated Code Suggestions [12.254014187965831]
  This paper conducts an empirical study using 66,329 industrial developer-AI interactions from a large technology company.<n>We analyze features that are significantly different between accepted code suggestions and rejected ones.<n>We introduce CSAP (Code Suggestion Acceptance Prediction) to predict whether a developer will accept the code suggestion before it is displayed.
  arXiv  Detail & Related papers  (2026-01-29T08:14:24Z)
• Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization [82.29178197694819]
  We derive and evaluate development-specific prompt optimization guidelines.<n>We use an iterative, test-driven approach to automatically refine code generation prompts.<n>We conduct an assessment with 50 practitioners, who report their usage of the elicited prompt improvement patterns.
  arXiv  Detail & Related papers  (2026-01-19T15:01:42Z)
• QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code [52.66657751895655]
  Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation.<n>This paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation.<n>We propose a self-evolving prompt optimization method that enables LLMs to evolve their internal prompt strategies.
  arXiv  Detail & Related papers  (2025-11-03T03:20:26Z)
• Building an Open AIBOM Standard in the Wild [5.0578463541356635]
  This paper presents a detailed experience report on the development of the AI Bill of Materials AIBOM specification.<n>We document a global, multi-stakeholder effort involving over 90 contributors and structured AR cycles.<n>The resulting specification was validated through four complementary approaches.
  arXiv  Detail & Related papers  (2025-10-08T14:32:31Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code [76.80306464249217]
  We propose TeaR, which aims at teaching LLMs to reason better.<n>TeaR leverages careful data curation and reinforcement learning to guide models in discovering optimal reasoning paths through code-related tasks.<n>We conduct extensive experiments using two base models and three long-CoT distillation models, with model sizes ranging from 1.5 billion to 32 billion parameters, and across 17 benchmarks spanning Math, Knowledge, Code, and Logical Reasoning.
  arXiv  Detail & Related papers  (2025-07-10T07:34:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.