Android Instrumentation Testing in Continuous Integration: Practices, Patterns, and Performance

• URL: http://arxiv.org/abs/2604.03438v1
• Date: Fri, 03 Apr 2026 20:24:19 GMT
• Title: Android Instrumentation Testing in Continuous Integration: Practices, Patterns, and Performance
• Abstract summary: We study how open-source Android apps run instrumentation tests in continuous integration.<n>Only about one in ten repositories run instrumentation tests in CI.<n>We find that teams often change approaches to expand test coverage.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Android instrumentation tests (end-to-end tests that run on a device or emulator) can catch problems that simpler tests miss. However, running these tests automatically in continuous integration (CI) is often difficult because emulator setup is fragile and configurations tend to drift over time. We study how open-source Android apps run instrumentation tests in CI by analyzing 4,518 repositories that use CI (snapshot: Aug. 10, 2025). We examine CI workflow files, scripts, and build configurations to identify cases where device setup is defined in Gradle (e.g., Gradle Managed Devices). Our results answer three questions about adoption, evolution, and outcomes. First, only about one in ten repositories (481/4,518; 10.6%) run instrumentation tests in CI, typically using either reusable community components or repository-specific custom scripts to set up emulators. Second, these setups usually stay the same over time; when changes happen, projects tend to move from custom scripts toward reusable community components. Third, we study why projects change their CI setup by analyzing their commits, pull requests, and issue messages. We evaluate how different setup styles perform using GitHub Actions run- and step-level metadata (e.g., outcomes, duration, reruns, and queue delay). We find that teams often change approaches to expand test coverage, and that each approach fits different needs: community-based setups are typically the most reliable and efficient for everyday checks on new code, third-party device labs suit scheduled regression testing but can be costlier and fail more often, and custom scripting provides flexibility but is associated with more reruns.

Related papers

• AppEval: A Unified Benchmark for LLM-Based Mobile Application Repair in ArkTS, Swift, and Kotlin [25.775485279136763]
  AppEval is a benchmark and native-toolchain evaluation framework for mobile application repair.<n>We present AppEval, a benchmark and native-toolchain evaluation framework for mobile application repair across HarmonyOS/ArkTS, iOS/Swift, and Android/Kotlin.
  arXiv  Detail & Related papers  (2026-08-19T06:30:34Z)
• DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testing [3.0812793454543375]
  DragonCrawl is an AI-driven mobile testing system for continuous regression testing.<n>It validates specific user flows on every code change, blocking commits that break critical functionality.<n>It achieves 91.6% pass rate on iOS and 92.2% on Android across 1,013 automated tests running continuously in CI/CD pipelines.
  arXiv  Detail & Related papers  (2026-07-30T18:17:31Z)
• Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution [20.606877071567958]
  We present TEBench, the first project-level benchmark for test evolution.<n>We construct TEBench through a four-stage pipeline over Defects4J projects.<n>We evaluate seven configurations spanning three industrial agent frameworks.
  arXiv  Detail & Related papers  (2026-05-07T12:31:09Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• Augmenting unit test suites from integration tests [0.3437656066916039]
  We propose a method that employs static and dynamic analysis for augmenting a test suite with automatically generated unit tests.<n>The method is most suitable for test suites where the stratification of unit, integration and system tests does not conform to the recommended test pyramid structure.<n>We showcase and empirically evaluate the proposed method in the Node.js platform, although it can be ported and adapted to other languages and platforms.
  arXiv  Detail & Related papers  (2026-04-19T15:58:55Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• EnvBench: A Benchmark for Automated Environment Setup [76.02998475135824]
  Large Language Models have enabled researchers to focus on practical repository-level tasks in software engineering domain.<n>Existing studies on environment setup introduce innovative agentic strategies, but their evaluation is often based on small datasets.<n>To address this gap, we introduce a comprehensive environment setup benchmark EnvBench.
  arXiv  Detail & Related papers  (2025-03-18T17:19:12Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• CI/CD Configuration Practices in Open-Source Android Apps: An Empirical Study [0.1433758865948252]
  Continuous Integration and Continuous Delivery (CI/CD) is a well-established practice that automatically builds, tests, packages, and deploys software systems.<n>Mobile apps have distinct characteristics with respect to CI/CD practices, such as testing on various emulators and deploying to app stores.<n>We conduct an empirical study on CI/CD practices in 2,557 Android apps adopting four popular CI/CD services.
  arXiv  Detail & Related papers  (2024-11-09T05:46:43Z)
• Observation-based unit test generation at Meta [52.4716552057909]
  TestGen automatically generates unit tests, carved from serialized observations of complex objects, observed during app execution. TestGen has landed 518 tests into production, which have been executed 9,617,349 times in continuous integration, finding 5,702 faults. Our evaluation reveals that, when carving its observations from 4,361 reliable end-to-end tests, TestGen was able to generate tests for at least 86% of the classes covered by end-to-end tests.
  arXiv  Detail & Related papers  (2024-02-09T00:34:39Z)
• Asynchronous Integration of Real-Time Simulators for HIL-based Validation of Smart Grids [0.08796261172196743]
  This paper explores the possibilities that are opened in terms of testing by the integration of a real-time simulator into co-simulation environments. Smart grid applications would typically include a relatively large number of physical devices, software components, as well as communication technology, all working hand in hand.
  arXiv  Detail & Related papers  (2023-09-14T11:44:21Z)
• Nirikshak: A Clustering Based Autonomous API Testing Framework [0.0]
  Nirikshak is a self-reliant testing framework for REST API testing. It achieves level 2 of autonomy in executing REST API testing procedures. Nirikshak is publicly available as an open-source software for the community at https://github.com/yashmahalwal/nirikshak.
  arXiv  Detail & Related papers  (2021-12-15T18:05:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.