Can LLMs Test Terminal User Interfaces?

• URL: http://arxiv.org/abs/2608.03743v1
• Date: Tue, 04 Aug 2026 14:36:44 GMT
• Title: Can LLMs Test Terminal User Interfaces?
• Abstract summary: Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment.<n>Only 12% of test code exercises the interface, and 45% of those tests never send input.<n>We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript.
• Score: 18.37827501923231
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.

Related papers

• How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection [41.369102837712795]
  Flaky tests pass and fail on the same code version, weakening the signal of test results.<n>Code-based flakiness detectors report strong benchmark results, yet their use in practice remains limited.<n>We argue that flakiness is not a static property of test code, which often lacks the information needed to decide whether a test is flaky.
  arXiv  Detail & Related papers  (2026-07-10T12:24:25Z)
• Covering Human Action Space for Computer Use: Data Synthesis and Benchmark [59.01879944842542]
  Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude.<n>Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust.<n>We propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions.
  arXiv  Detail & Related papers  (2026-05-12T17:59:58Z)
• Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution [20.606877071567958]
  We present TEBench, the first project-level benchmark for test evolution.<n>We construct TEBench through a four-stage pipeline over Defects4J projects.<n>We evaluate seven configurations spanning three industrial agent frameworks.
  arXiv  Detail & Related papers  (2026-05-07T12:31:09Z)
• Improving Random Testing via LLM-powered UI Tarpit Escaping for Mobile Apps [5.901783684504153]
  We introduce LLM-powered random GUI Testing, a novel hybrid testing approach to mitigating UI tarpits during random testing.<n>Our approach monitors UI similarity to identify tarpits and query LLMs to suggest promising events for escaping the encountered tarpits.<n> HybridMonkey and HybridDroidbot outperform all baselines, achieving average coverage improvements of 54.8% and 44.8%, respectively.
  arXiv  Detail & Related papers  (2026-04-08T07:28:26Z)
• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis [57.40527331817245]
  Test oracle generation in non-regression testing is a longstanding challenge in software engineering.<n>We introduce Nexus, a novel multi-agent framework to address this challenge.
  arXiv  Detail & Related papers  (2025-10-30T12:20:25Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective [51.30005925128432]
  evaluating test cases automatically generated by Large Language Models (LLMs) is a critical yet challenging task.<n>Existing benchmarks suffer from high computational costs, score inflation, and a bias towards trivial bugs over rare, critical faults.<n>We introduce a framework that formalizes benchmark construction as finding an optimal diagnostic basis in a binary code-test matrix.
  arXiv  Detail & Related papers  (2025-10-09T18:29:24Z)
• Phi-Ground Tech Report: Advancing Perception in GUI Grounding [55.44475003199647]
  textbfPhi-Ground model family achieves state-of-the-art performance across all five grounding benchmarks.<n>textittextbf43.2 on ScreenSpot-pro and textittextbf27.2 on UI-Vision.
  arXiv  Detail & Related papers  (2025-07-31T17:59:09Z)
• TESTEVAL: Benchmarking Large Language Models for Test Case Generation [15.343859279282848]
  We propose TESTEVAL, a novel benchmark for test case generation with large language models (LLMs)<n>We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage.<n>We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs.
  arXiv  Detail & Related papers  (2024-06-06T22:07:50Z)
• An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation [3.9762912548964864]
  This paper presents a large-scale empirical evaluation on the effectiveness of Large Language Models for automated unit test generation. We implement our approach in TestPilot, a test generation tool for JavaScript that automatically generates unit tests for all API functions in an npm package. We find that 92.8% of TestPilot's generated tests have no more than 50% similarity with existing tests.
  arXiv  Detail & Related papers  (2023-02-13T17:13:41Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.