DeepFix: Debugging and Fixing Machine Learning Workflow using Agentic AI

• URL: http://arxiv.org/abs/2603.14099v1
• Date: Sat, 14 Mar 2026 20:00:16 GMT
• Title: DeepFix: Debugging and Fixing Machine Learning Workflow using Agentic AI
• Abstract summary: DeepFix is a tool for automated testing of the entire machine learning pipeline using an agentic AI framework.<n>Our testing approach first leverages Deepchecks to test the ML software for any potential bugs, and thereafter, uses an agentic AI-based approach to generate a detailed bug report.<n>This includes a ranking, based on the severity of the found bugs, along with their explanations, which can be interpreted easily by any non-data science experts.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, machine learning (ML) based software systems are increasingly deployed in several critical applications, yet systematic testing of their behavior remains challenging due to complex model architectures, large input spaces, and evolving deployment environments. Existing testing approaches often rely on generating test cases based on given requirements, which often fail to reveal critical bugs of modern ML models due to their complex nature. Most importantly, such approaches, although they can be used to detect the presence of specific failures in the ML software, they hardly provide any message as to how to fix such errors. To tackle this, in this paper, we present DeepFix, a tool for automated testing of the entire ML pipeline using an agentic AI framework. Our testing approach first leverages Deepchecks to test the ML software for any potential bugs, and thereafter, uses an agentic AI-based approach to generate a detailed bug report. This includes a ranking, based on the severity of the found bugs, along with their explanations, which can be interpreted easily by any non-data science experts and most importantly, also provides possible ways to fix these bugs. Additionally, DeepFix supports several types of ML software systems and can be integrated easily to any ML workflow, enabling continuous testing throughout the development lifecycle. We discuss our already validated cases as well as some planned validations designed to demonstrate how the agentic testing process can reveal hidden failure modes that remain undetected by conventional testing methods. A 5-minute screencast demonstrating the tool's core functionality is available at https://youtu.be/WfwZmFcQgBQ.

Related papers

• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• VIBEPASS: Can Vibe Coders Really Pass the Vibe Check? [46.85901599242161]
  We evaluate two coupled tasks: emphFault-Triggering Test Generation (FT-Test) and emphFault-targeted Program Repair (FPR)<n>We find that fault-targeted reasoning does not scale with general coding ability.<n>Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines.
  arXiv  Detail & Related papers  (2026-03-16T21:14:28Z)
• Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software [3.853925623717688]
  This paper presents an automated pipeline to detect flaky-test-related issues and pull requests in quantum software repositories.<n>We expand an existing quantum flaky test dataset and evaluate the capability of Large Language Models for flakiness classification and root-cause identification.<n>The best-performing model, Google Gemini, achieves an F1-score of 0.9420 for flakiness detection and 0.9643 for root-cause identification.
  arXiv  Detail & Related papers  (2026-03-09T23:57:55Z)
• Automated structural testing of LLM-based agents: methods, framework, and case studies [0.05254956925594667]
  LLM-based agents are rapidly being adopted across diverse domains.<n>Current testing approaches focus on acceptance-level evaluation from the user's perspective.<n>We present methods to enable structural testing of LLM-based agents.
  arXiv  Detail & Related papers  (2026-01-25T11:52:30Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• Using ML filters to help automated vulnerability repairs: when it helps and when it doesn't [5.10123605644148]
  Testing requires typically a costly process of building the application while ML models can be used to quickly classify patches.<n>We propose to use an ML model as a preliminary filter of candidate patches which is put in front of a traditional filter based on testing.
  arXiv  Detail & Related papers  (2025-04-09T16:39:09Z)
• Design choices made by LLM-based test generators prevent them from finding bugs [0.850206009406913]
  This paper critically examines whether recent LLM-based test generation tools, such as Codium CoverAgent and CoverUp, can effectively find bugs or unintentionally validate faulty code.<n>Using real human-written buggy code as input, we evaluate these tools, showing how LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests.
  arXiv  Detail & Related papers  (2024-12-18T18:33:26Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation [96.71370747681078]
  We introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate.
  arXiv  Detail & Related papers  (2023-10-05T04:06:12Z)
• Detecting Faults during Automatic Screwdriving: A Dataset and Use Case of Anomaly Detection for Automatic Screwdriving [80.6725125503521]
  Data-driven approaches, using Machine Learning (ML) for detecting faults have recently gained increasing interest. We present a use case of using ML models for detecting faults during automated screwdriving operations.
  arXiv  Detail & Related papers  (2021-07-05T11:46:00Z)
• Mutation Testing framework for Machine Learning [0.0]
  Failure of Machine Learning Models can lead to severe consequences in terms of loss of life or property. Developers, scientists, and ML community around the world, must build a highly reliable test architecture for critical ML application. This article provides an insight journey of Machine Learning Systems (MLS) testing, its evolution, current paradigm and future work.
  arXiv  Detail & Related papers  (2021-02-19T18:02:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.