Learning Correct Behavior from Examples: Validating Sequential Execution in Autonomous Agents

• URL: http://arxiv.org/abs/2605.03159v1
• Date: Mon, 04 May 2026 21:07:56 GMT
• Title: Learning Correct Behavior from Examples: Validating Sequential Execution in Autonomous Agents
• Abstract summary: Traditional testing approaches require manual specification, exact sequence matching, or thousands of training examples.<n>We present a novel algorithm that automatically learns correct behavior from just 2-10 passing execution traces.<n>In controlled experiments, our system achieved high accuracy in detecting product bugs and false successes.
• Score: 6.9734639631942015
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As autonomous agents become increasingly sophisticated, validating their sequential behavior presents a significant challenge. Traditional testing approaches require manual specification, exact sequence matching, or thousands of training examples. We present a novel algorithm that automatically learns correct behavior from just 2-10 passing execution traces and validates new executions against this learned model. Our approach combines dominator analysis from compiler theory with multimodal large language model-powered semantic understanding to identify essential states and handle non-deterministic behavior. The system constructs a generalized ground truth model using Prefix Tree Acceptors, merges traces through multi-tiered equivalence detection, and validates new executions via topological subsequence matching. In controlled experiments, our system achieved high accuracy in detecting product bugs and false successes using only 3 training traces. This approach provides explainable validation results with coverage metrics and works across diverse domains including UI testing, code generation, and robotic processes.

Related papers

• The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning [54.67958855362658]
  We propose a dynamic framework that stress tests unlearning using complex structured queries.<n>Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains.<n>Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets.
  arXiv  Detail & Related papers  (2026-03-11T19:51:33Z)
• Scaling Agentic Verifier for Competitive Coding [66.11758166379092]
  Large language models (LLMs) have demonstrated strong coding capabilities but still struggle to solve competitive programming problems correctly in a single attempt.<n>Execution-based re-ranking offers a promising test-time scaling strategy, yet existing methods are constrained by either difficult test case generation or inefficient random input sampling.<n>We propose Agentic Verifier, an execution-based agent that actively reasons about program behaviors and searches for highly discriminative test inputs.
  arXiv  Detail & Related papers  (2026-02-04T06:30:40Z)
• Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking [54.43083499412643]
  Test-time algorithms that combine the generative power of language models with process verifiers offer a promising lever for eliciting new reasoning capabilities.<n>We introduce a new process-guided test-time sampling algorithm, VGB, which uses theoretically grounded backtracking to achieve provably better robustness to verifier errors.
  arXiv  Detail & Related papers  (2025-10-03T16:21:14Z)
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [58.05959902776133]
  We introduce Single-Pass.<n>with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation.<n>We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP)<n>On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $sim$16% of training samples compared to human-labeled and other synthetically trained baselines.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)
• Consistency-based Abductive Reasoning over Perceptual Errors of Multiple Pre-trained Models in Novel Environments [5.5855749614100825]
  This paper addresses the hypothesis that leveraging multiple pre-trained models can mitigate this recall reduction.<n>We formulate the challenge of identifying and managing conflicting predictions from various models as a consistency-based abduction problem.<n>Our results validate the use of consistency-based abduction as an effective mechanism to robustly integrate knowledge from multiple imperfect models in challenging, novel scenarios.
  arXiv  Detail & Related papers  (2025-05-25T23:17:47Z)
• A Case Study on Model Checking and Runtime Verification for Awkernel [0.7199733380797578]
  It is difficult for humans to manually review concurrent behaviors or to write test cases covering all possible executions.<n>This paper proposes a development method for concurrent software, such as schedulers.
  arXiv  Detail & Related papers  (2025-03-12T11:27:45Z)
• A Progressive Transformer for Unifying Binary Code Embedding and Knowledge Transfer [15.689556592544667]
  We introduce ProTST, a novel transformer-based methodology for binary code embedding.<n>ProTST employs a hierarchical training process based on a unique tree-like structure.<n>Results show that ProTST yields an average validation score (F1, MRR, and Recall@1) improvement of 14.8% compared to traditional two-stage training.
  arXiv  Detail & Related papers  (2024-12-15T13:04:29Z)
• Engineering the Neural Automatic Passenger Counter [0.0]
  We explore and exploit various aspects of machine learning to increase reliability, performance, and counting quality. We show how aggregation techniques such as ensemble quantiles can reduce bias, and we give an idea of the overall spread of the results.
  arXiv  Detail & Related papers  (2022-03-02T14:56:11Z)
• A Novel Anomaly Detection Algorithm for Hybrid Production Systems based on Deep Learning and Timed Automata [73.38551379469533]
  DAD:DeepAnomalyDetection is a new approach for automatic model learning and anomaly detection in hybrid production systems. It combines deep learning and timed automata for creating behavioral model from observations. The algorithm has been applied to few data sets including two from real systems and has shown promising results.
  arXiv  Detail & Related papers  (2020-10-29T08:27:43Z)
• Document Ranking with a Pretrained Sequence-to-Sequence Model [56.44269917346376]
  We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words" Our approach significantly outperforms an encoder-only model in a data-poor regime.
  arXiv  Detail & Related papers  (2020-03-14T22:29:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.