Scaling Mobile Chaos Testing with AI-Driven Test Execution

• URL: http://arxiv.org/abs/2602.06223v1
• Date: Thu, 05 Feb 2026 22:01:50 GMT
• Title: Scaling Mobile Chaos Testing with AI-Driven Test Execution
• Abstract summary: Mobile applications in large-scale distributed systems are susceptible to backend service failures.<n>Traditional chaos engineering approaches cannot scale mobile testing due to explosion of flows, locations, and failure scenarios.<n>We present an automated mobile chaos testing system that integrates DragonCrawl, an LLM-based mobile testing platform, with uHavoc, a service-level fault injection system.
• Score: 2.7786234871633995
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mobile applications in large-scale distributed systems are susceptible to backend service failures, yet traditional chaos engineering approaches cannot scale mobile testing due to the combinatorial explosion of flows, locations, and failure scenarios that need validation. We present an automated mobile chaos testing system that integrates DragonCrawl, an LLM-based mobile testing platform, with uHavoc, a service-level fault injection system. The key insight is that adaptive AI-driven test execution can navigate mobile applications under degraded backend conditions, eliminating the need to manually write test cases for each combination of user flow, city, and failure type. Since Q1 2024, our system has executed over 180,000 automated chaos tests across 47 critical flows in Uber's Rider, Driver, and Eats applications, representing approximately 39,000 hours of manual testing effort that would be impractical at this scale. We identified 23 resilience risks, with 70% being architectural dependency violations where non-critical service failures degraded core user flows. Twelve issues were severe enough to prevent trip requests or food orders. Two caused application crashes detectable only through mobile chaos testing, not backend testing alone. Automated root cause analysis reduced debugging time from hours to minutes, achieving 88% precision@5 in attributing mobile failures to specific backend services. This paper presents the system design, evaluates its performance under fault injection (maintaining 99% test reliability), and reports operational experience demonstrating that continuous mobile resilience validation is achievable at production scale.

Related papers

• DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testing [3.0812793454543375]
  DragonCrawl is an AI-driven mobile testing system for continuous regression testing.<n>It validates specific user flows on every code change, blocking commits that break critical functionality.<n>It achieves 91.6% pass rate on iOS and 92.2% on Android across 1,013 automated tests running continuously in CI/CD pipelines.
  arXiv  Detail & Related papers  (2026-07-30T18:17:31Z)
• Practical Limits of Autonomous Test Repair: A Multi-Agent Case Study with LLM-Driven Discovery and Self-Correction [0.0]
  We present an industrial case study of a multi-agent autonomous testing system evaluated using anonymized execution data from a production-like enterprise UI testing prototype.<n>The system evolves from human-directed testing toward High-autonomy feature discovery and test execution.<n>Our findings show that unrestricted autonomy leads to unstable and often misleading outcomes, while constrained autonomy transforms such systems into operationally viable.
  arXiv  Detail & Related papers  (2026-05-02T14:39:55Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• ProbeLLM: Automating Principled Diagnosis of LLM Failures [89.44131968886184]
  We propose ProbeLLM, a benchmark-agnostic automated probing framework that elevates weakness discovery from individual failures to structured failure modes.<n>By restricting probing to verifiable test cases and leveraging tool-augmented generation and verification, ProbeLLM grounds failure discovery in reliable evidence.
  arXiv  Detail & Related papers  (2026-02-13T14:33:13Z)
• Cast: Automated Resilience Testing for Production Cloud Service Systems [38.54479293660192]
  We present Cast, an automated, end-to-end framework for microservice resilience testing in production.<n>It achieves high test fidelity by replaying production traffic against a comprehensive library of application-level faults.<n>Cast has been adopted by many service teams to proactively address resilience vulnerabilities.
  arXiv  Detail & Related papers  (2026-02-01T02:29:25Z)
• Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models [1.2744523252873348]
  FlaXifyer is a few-shot learning approach for predicting intermittent job failure categories using pre-trained language models.<n>LogSift is an interpretability technique that identifies influential log statements in under one second.<n> Evaluation on 2,458 job failures from TELUS demonstrates that FlaXifyer and LogSift enable effective automated triage, accelerate failure diagnosis, and pave the way towards the automated resolution of intermittent job failures.
  arXiv  Detail & Related papers  (2026-01-29T19:34:34Z)
• Misbehavior Forecasting for Focused Autonomous Driving Systems Testing [2.7733556309376692]
  Existing bug-finding techniques are either unreliable or expensive.<n>We propose Foresee, a technique that identifies near misses using a misbehavior forecaster.<n>Foresee performs local fuzzing in the neighborhood of each candidate near miss to surface previously unknown failures.
  arXiv  Detail & Related papers  (2025-12-21T17:17:49Z)
• An Automated Blackbox Noncompliance Checker for QUIC Server Implementations [2.9248916859490173]
  QUICtester is an automated approach for uncovering non-compliant behaviors in the ratified QUIC protocol implementations (RFC 9000/).<n>We used QUICtester to analyze 186 learned models from 19 QUIC implementations under the five security settings and discovered 55 implementation errors.
  arXiv  Detail & Related papers  (2025-05-19T04:28:49Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures [6.824747267214373]
  Flaky tests produce inconsistent outcomes without code changes.<n>Developers spend 1.28% of their time repairing flaky tests at a monthly cost of $2,250.<n>We show that flaky tests often exist in clusters, with co-occurring failures that share the same root causes, which we call systemic flakiness.
  arXiv  Detail & Related papers  (2025-04-23T14:51:23Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• Neural Embeddings for Web Testing [49.66745368789056]
  Existing crawlers rely on app-specific, threshold-based, algorithms to assess state equivalence. We propose WEBEMBED, a novel abstraction function based on neural network embeddings and threshold-free classifiers. Our evaluation on nine web apps shows that WEBEMBED outperforms state-of-the-art techniques by detecting near-duplicates more accurately.
  arXiv  Detail & Related papers  (2023-06-12T19:59:36Z)
• SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning [62.997667081978825]
  Testing video games is an increasingly difficult task as traditional methods fail to scale with growing software systems. We present SUPERNOVA, a system responsible for test selection and defect prevention while also functioning as an automation hub. The direct impact of this has been observed to be a reduction in 55% or more testing hours for an undisclosed sports game title.
  arXiv  Detail & Related papers  (2022-03-10T00:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.