Do you dare to try Test-Driven Forensics? Increasing Trust in Desktop Forensics with ADARE

• URL: http://arxiv.org/abs/2605.28476v1
• Date: Wed, 27 May 2026 13:38:06 GMT
• Title: Do you dare to try Test-Driven Forensics? Increasing Trust in Desktop Forensics with ADARE
• Abstract summary: We present test-driven forensics, a practical approach that treats forensic expectations as executable specifications.<n>We implement the methodology in ADARE, an open-source framework that runs controlled experiments in virtual machines.<n>A 25-version regression study of Autopsy reveals substantial, largely undocumented changes in exported report outputs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Digital forensic relies on validated tools and established procedures, yet the underlying operating systems, applications, and analysis tools evolve rapidly. This evolution can cause artifact behavior and tool outputs to drift, silently degrading repeatability and confidence in long-lived forensic interpretations. We present test-driven forensics, a practical approach that treats forensic expectations as executable specifications: expected artifacts and expected tool outputs are encoded as tests that can be rerun across versions to detect regressions. Crucially, our approach also enables State Transition Testing, validating the system's expected state after each user action rather than only performing post-mortem checks on a final disk image; this supports causal attribution and makes transient behavior testable. We implement the methodology in ADARE, an open-source framework that runs controlled experiments in virtual machines and simulates realistic user activity via computer-vision-guided GUI automation. ADARE includes a companion web platform for sharing experiments, environments, and results to facilitate independent reruns and peer verification. We evaluate ADARE in five case studies spanning artifact research and tool validation. In particular, a 25-version regression study of Autopsy reveals substantial, largely undocumented changes in exported report outputs, demonstrating how executable tests make drift measurable and reproducible at scale.

Related papers

• Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills [14.44743150931186]
  Tests increasingly participate in the decisions made by large language models and software engineering agents.<n>They specify intended behavior, guide program construction and repair, select candidates, constrain transformations, and provide execution evidence for software analysis.<n>These uses draw on test-driven development, yet differ substantially in test order, oracle availability, editable artifacts, and the role of execution.
  arXiv  Detail & Related papers  (2026-09-10T06:20:22Z)
• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• Context-Aware Generative AI for Automated Telecom Test Script Generation [0.0024792470319550896]
  Existing solutions generate static test suites against a snapshot of the system.<n>As code, configurations, topologies, and key performance indicators (KPIs) evolve, these tests quickly become outdated or misaligned with the live system.<n>This paper presents a context-aware generative AI framework for automated telecom test script generation.
  arXiv  Detail & Related papers  (2026-06-19T06:44:31Z)
• Claw AI Lab: An Autonomous Multi-Agent Research Team [73.07229229907435]
  We present Claw AI Lab, a lab-native autonomous research platform.<n>Rather than centering the system around a single agent or a fixed serial workflow, we allow users to instantiate a full research team from one prompt.<n>Claw-Code Harness connects locals, datasets, and checkpoints to runnable experiments and feeds execution artifacts back into the research loop.
  arXiv  Detail & Related papers  (2026-05-21T16:02:53Z)
• Sibyl-AutoResearch: Autonomous Research Needs Self-Evolving Trial-and-Error Harnesses, Not Paper Generators [37.075000666622074]
  We introduce Sibyl-AutoResearch, a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses.<n>A harness lets agents run bounded trials, preserve positive and negative outcomes, and route lessons into later planning, validation, claim scope, scheduling, critique, writing, and repair.<n>We implement the framework in SIBYL, a file-backed autonomous research system that exposes the state, roles, memory, gates, and artifact traces needed to inspect these conversion paths.
  arXiv  Detail & Related papers  (2026-05-21T11:29:08Z)
• Cochise: A Reference Harness for Autonomous Penetration Testing [1.3124479769761592]
  We present cochise, a 597 LOC Python reference harness for autonomous penetration-testing.<n>Cochise connects an LLM-driven agent to a Linux host over SSH and supports controlled target environments.<n>We release replay and analysis tools: (i) cochise-replay for offline visualization of captured runs, (ii) cochise-analyzelogs and cochise-analyze-a-graphs for cost, token duration, and compromise analysis.
  arXiv  Detail & Related papers  (2026-05-12T07:28:12Z)
• Practical Limits of Autonomous Test Repair: A Multi-Agent Case Study with LLM-Driven Discovery and Self-Correction [0.0]
  We present an industrial case study of a multi-agent autonomous testing system evaluated using anonymized execution data from a production-like enterprise UI testing prototype.<n>The system evolves from human-directed testing toward High-autonomy feature discovery and test execution.<n>Our findings show that unrestricted autonomy leads to unstable and often misleading outcomes, while constrained autonomy transforms such systems into operationally viable.
  arXiv  Detail & Related papers  (2026-05-02T14:39:55Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Scaling Agentic Verifier for Competitive Coding [66.11758166379092]
  Large language models (LLMs) have demonstrated strong coding capabilities but still struggle to solve competitive programming problems correctly in a single attempt.<n>Execution-based re-ranking offers a promising test-time scaling strategy, yet existing methods are constrained by either difficult test case generation or inefficient random input sampling.<n>We propose Agentic Verifier, an execution-based agent that actively reasons about program behaviors and searches for highly discriminative test inputs.
  arXiv  Detail & Related papers  (2026-02-04T06:30:40Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Towards Automatic Generation of Amplified Regression Test Oracles [44.45138073080198]
  We propose a test oracle derivation approach to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour.
  arXiv  Detail & Related papers  (2023-07-28T12:38:44Z)
• Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representations [5.439020425819001]
  We propose a hybrid machine learning architecture that simultaneously employs multiple deep learning models. We report an improved detection rate, above the capabilities of the current state-of-the-art model.
  arXiv  Detail & Related papers  (2022-08-20T05:30:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.