In-IDE Toolkit for Developers of AI-Based Features

• URL: http://arxiv.org/abs/2605.14612v1
• Date: Thu, 14 May 2026 09:28:14 GMT
• Title: In-IDE Toolkit for Developers of AI-Based Features
• Abstract summary: We present the AI Toolkit plugin for JetBrains, which brings tracing and evaluation directly into the Run/ Debug loop.<n>We detail the design and implementation of the AI Agentsger and AI Evaluation, report initial adoption telemetry, and outline next steps to broaden framework coverage and scale evaluations.
• Score: 0.24629531282150877
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI-enabled features built on LLMs and agentic workflows are difficult to test, debug, and reproduce, especially for product-focused software engineers without a machine learning background. We present the AI Toolkit plugin for JetBrains IDEs, which brings tracing and evaluation directly into the Run/Debug loop. A mixed methods study with practitioners presents three consistent needs: (1) make evaluation regular and repeatable, (2) expose traces at the moment of execution, and (3) minimize setup and context switching. Guided by these needs, the AI Toolkit introduces an IDE-native workflow: run-triggered trace capture; immediate, hierarchical inspection; one-click "Add to Dataset" from traces; and unit-test-like evaluations with pluggable metrics. The first release in PyCharm shows promising early signals - strong conversion when promoted at Run, sustained usage among those who capture traces, and low churn - suggesting that IDE-native observability lowers activation energy and helps developers adopt disciplined practices. We detail the design and implementation of the AI Agents Debugger and AI Evaluation, report initial adoption telemetry, and outline next steps to broaden framework coverage and scale evaluations. Together, these results indicate that integrating AI observability and evaluation into everyday IDE workflows can make modern AI development accessible to non-ML specialists while preserving software-engineering practices.

Related papers

• IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks [0.37823923040445995]
  We present a Dockerized test harness that goes beyond raw terminal execution.<n>We provide high-level abstractions for search, structured file editing, and tools for testing full-stack applications.
  arXiv  Detail & Related papers  (2026-01-28T02:06:37Z)
• DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle [84.01703913780946]
  Handling the DevOps cycle in real-world software requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions.<n>We introduce DevOps-Gym, the first end-to-end benchmark for evaluating AI agents across core DevOps tasks.
  arXiv  Detail & Related papers  (2026-01-27T18:43:46Z)
• AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development [12.50615284537175]
  Large language model (LLM) based coding agents increasingly act as autonomous contributors that generate and merge pull requests.<n>We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls.
  arXiv  Detail & Related papers  (2026-01-20T04:51:56Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• MIRA: Empowering One-Touch AI Services on Smartphones with MLLM-based Instruction Recommendation [61.19099947706954]
  This paper introduces MIRA, a pioneering framework for task instruction recommendation.<n>With MIRA, users can long-press on images or text objects to receive contextually relevant instruction recommendations for executing AI tasks.<n> MIRA has demonstrated substantial improvements in the accuracy of instruction recommendation.
  arXiv  Detail & Related papers  (2025-09-17T07:43:14Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• How Software Engineers Engage with AI: A Pragmatic Process Model and Decision Framework Grounded in Industry Observations [1.516251872371896]
  GitHub Copilot and ChatGPT have given rise to "vibe coding"<n>This paper presents two complementary contributions.<n>First, a pragmatic process model capturing real-world AI-assisted SE activities, including prompt design, inspection, fallback, and refinement.<n>Second, a 2D decision framework that could help developers reason about trade-offs between effort saved and output quality.
  arXiv  Detail & Related papers  (2025-07-23T21:00:21Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Disrupting Test Development with AI Assistants [1.024113475677323]
  Generative AI-assisted coding tools like GitHub Copilot, ChatGPT, and Tabnine have significantly transformed software development. This paper analyzes how these innovations impact productivity and software test development metrics.
  arXiv  Detail & Related papers  (2024-11-04T17:52:40Z)
• MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images [49.664220687980006]
  The lack of annotated datasets is a major bottleneck for training new task-specific supervised machine learning models. We present MONAI Label, a free and open-source framework that facilitates the development of applications based on artificial intelligence (AI) models.
  arXiv  Detail & Related papers  (2022-03-23T12:33:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.