From Natural Language to Executable Properties for Property-based Testing of Mobile Apps

• URL: http://arxiv.org/abs/2603.21263v1
• Date: Sun, 22 Mar 2026 14:47:25 GMT
• Title: From Natural Language to Executable Properties for Property-based Testing of Mobile Apps
• Abstract summary: We propose a novel structured property synthesis approach that automatically translates property descriptions in natural language into executable properties.<n>Our approach decomposes the problem into UI semantic grounding and executable property synthesis.<n>We show that iPBT reduces the time required to write executable properties by 56%, suggesting substantially lower manual effort.
• Score: 15.233172989165274
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Property-based testing (PBT) is a popular software testing methodology and is effective in validating the functionality of mobile applications (apps for short). However, its adoption in practice remains limited, largely due to the manual effort and technical expertise required to specify executable properties. In this experience paper, we propose a novel structured property synthesis approach that automatically translates property descriptions in natural language into executable properties, and implement it in a tool named iPBT. Our approach decomposes the problem into UI semantic grounding and executable property synthesis. It first builds an enriched widget context via multimodal LLMs to align visual elements with their functional semantics, and then uses an LLM with in-context learning to generate framework-specific executable properties. We evaluate iPBT with a closed-source LLM (GPT-4o) and an open-source LLM (DeepSeek-V3) on 124 diverse property descriptions derived from an existing benchmark dataset. iPBT achieves 95.2% (118/124) accuracy on both LLMs. Notably, an ablation study reveals that the enriched widget context contributes to an absolute improvement of up to 20.2% (from 75.0% to 95.2%). A user study with 10 participants demonstrates that iPBT reduces the time required to write executable properties by 56%, suggesting substantially lower manual effort. Furthermore, evaluations on 1,180 linguistically diverse variations demonstrate iPBT's robustness (87.6% accuracy), indicating its capability to handle varied expressions.

Related papers

• USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning [54.44143638529487]
  Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models.<n>We propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages.
  arXiv  Detail & Related papers  (2026-07-04T14:45:23Z)
• Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition [52.07624601400713]
  In-context learning (ICL) with large language models (LLMs) has emerged as a powerful alternative to fine-tuning for Named Entity Recognition (NER)<n>We conduct a comprehensive investigation of many-shot ICL for NER and explore its effectiveness in annotating and refining data for low-resource NER tasks.<n>Our experiments demonstrate that: (1) scaling to hundreds of in-context examples enables LLMs to match or even surpass the performance of fully supervised BERT models; and (2) using about one hundred human-labeled examples as demonstrations, many-shot in-context annotation can generate high-quality labeled
  arXiv  Detail & Related papers  (2026-06-20T05:39:59Z)
• Explainable Semantic Textual Similarity via Dissimilar Span Detection [53.32175252285023]
  We introduce the task of Dissimilar Span Detection (DSD), which aims to identify semantically differing spans between pairs of texts.<n>This can help users understand which particular words or tokens negatively affect the similarity score, or be used to improve performance in STS-dependent downstream tasks.
  arXiv  Detail & Related papers  (2026-03-22T11:32:31Z)
• Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI [4.357716629393195]
  Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts.<n>This study focuses on reducing labeling effort in ATD detection by combining keyword-based filtering with active learning and explainable AI.
  arXiv  Detail & Related papers  (2026-03-03T12:51:54Z)
• ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization [1.580774794371876]
  We present ARIAL, a framework that orchestrates specialized tools to achieve both precise answer extraction and reliable spatial grounding.<n>We evaluate ARIAL on four benchmarks (DocVQA, FUNSD, CORD, and SROIE) using both textual accuracy (ANLS) and spatial precision.<n>Our work demonstrates how agentic orchestration of specialized tools can simultaneously improve performance and interpretability.
  arXiv  Detail & Related papers  (2025-11-22T21:09:28Z)
• Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation [3.9189409002585567]
  Large language models (LLMs) have demonstrated strong performance on function-level code generation benchmarks.<n>We introduce a benchmark derived from real-world open-source repositories to evaluate generalization under practical conditions.<n>We examine how input specification completeness and retrieval-augmented generation affect class-level correctness across multiple state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-10-30T04:30:23Z)
• Active Authentication via Korean Keystrokes Under Varying LLM Assistance and Cognitive Contexts [3.091662349810693]
  We evaluate keystroke-based authentication in Korean across three realistic typing scenarios.<n>We show that the system maintains reliable performance across varying LLM usages and cognitive contexts.
  arXiv  Detail & Related papers  (2025-09-29T13:57:16Z)
• PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generation [11.350635014083812]
  PropertyGPT can generate comprehensive and high-quality properties, achieving an 80% recall compared to the ground truth.<n>PropertyGPT detected 26 CVEs/attack incidents out of 37 tested and also uncovered 12 zero-day vulnerabilities, leading to $8,256 in bug bounty rewards.
  arXiv  Detail & Related papers  (2024-05-04T06:28:27Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback [78.60644407028022]
  We introduce MINT, a benchmark that evaluates large language models' ability to solve tasks with multi-turn interactions. LLMs generally benefit from tools and language feedback, with performance gains of 1-8% for each turn of tool use. LLMs evaluated, supervised instruction-finetuning (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities.
  arXiv  Detail & Related papers  (2023-09-19T15:25:42Z)
• Can Large Language Models Write Good Property-Based Tests? [5.671039991090038]
  Property-based testing (PBT) is still relatively underused in real-world software. We investigate using modern language models to automatically synthesize PBTs using two prompting techniques. We find that with the best model and prompting approach, a valid and sound PBT can be synthesised in 2.4 samples on average.
  arXiv  Detail & Related papers  (2023-07-10T05:09:33Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)
• Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision [106.77639982059014]
  We present ConST-CL framework to effectively learn-temporally fine-grained representations. We first design a region-based self-supervised task which requires the model to learn to transform instance representations from one view to another guided by context features. We then introduce a simple design that effectively reconciles the simultaneous learning of both holistic and local representations.
  arXiv  Detail & Related papers  (2021-12-09T19:13:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.