Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

• URL: http://arxiv.org/abs/2608.05784v1
• Date: Thu, 06 Aug 2026 09:17:43 GMT
• Title: Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
• Abstract summary: We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline.<n>It segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows.<n>On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.

Related papers

• AgentWeave: Routing Before Reasoning for Efficient Function Calling in Tool-Rich Language Models [0.0]
  We introduce AgentWeave, a deterministic pre-inference routing layer that constructs a bounded model-visible action space.<n>Relative to all-tools exposure, AgentWeave presents 70.18% fewer tools, uses 61.70% fewer input tokens, and exhibits 50.95% lower mean local-model latency.
  arXiv  Detail & Related papers  (2026-08-24T10:38:08Z)
• Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation [2.1734195143282693]
  We propose reliability@k, the same estimator applied correctly, with n = independent rollouts and c = fully-passing rollouts per (task, agent) pair.<n>Motivated by evidence that functional correctness does not imply security safety, we additionally propose security-adjusted reliability@k.
  arXiv  Detail & Related papers  (2026-08-11T16:01:15Z)
• SAT-Edge-Agent: Hardware-in-the-Loop Edge-Agent Orchestration for Onboard Satellite Intelligence [0.0]
  We present SAT-Edge-Agent, a hardware-in-the-loop (HIL) edge-agent system deployed on a commercial ARM-based heterogeneous edge system-on-chip.<n>A browser workspace and FastAPI agent coordinate a local OpenAI-compatible language service with a project-internal YOLO-style oriented-object-detection endpoint.<n>Two fixed FAIR1M workloads, one single-image and one serial two-image request, were repeated 20 times each and completed 20/20 attempts.<n>Mean Full-Agent latency was 29.353 s and 60.937 s, with empirical P95 values of
  arXiv  Detail & Related papers  (2026-08-04T14:25:27Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• Bayesian-Agent: Posterior-Guided Skill Evolution for LLM Agent Harnesses [30.457068495035386]
  textbfBayesianAgent treats reusable skills and SOPs as hypotheses about whether a frozen model will succeed under a particular prompt, context, and harness environment.<n>Bayesian-Agent records verified trajectory evidence, maintains a feature-conditioned categorical posterior over each skill, and maps posterior state into inspectable actions such as patch, split, compress, retire, and explore.
  arXiv  Detail & Related papers  (2026-06-06T21:40:19Z)
• Scaling Self-Evolving Agents via Parametric Memory [69.96398842169002]
  Existing memory-augmented LLM agents store past experience exclusively in prompt space.<n>We introduce textttTMEM, a self-evolving parametric memory framework.<n>textttTMEM consistently outperforms summary-based and retrieval-based baselines across different model scales.
  arXiv  Detail & Related papers  (2026-06-03T07:18:31Z)
• Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study [0.0]
  The unit of analysis was the persistent human-agent environment: researcher, agent, memory layer, tools, repositories, scheduled jobs, specialized agent roles, and governance rules.<n>A strict May 2026 trajectory subset captured 627 model-related events and 73.95 million recorded tokens, of which 82.9% were cache reads.
  arXiv  Detail & Related papers  (2026-05-26T11:28:36Z)
• AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation [11.272830796781925]
  We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks.<n>Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass.<n>We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:00:57Z)
• Scaling Test-Time Compute for Agentic Coding [126.72747643609274]
  We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories.<n>Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes.<n>Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0.
  arXiv  Detail & Related papers  (2026-04-16T17:39:33Z)
• RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations [52.752467948588816]
  We propose RSAgent, an agentic Multimodal Large Language Model (MLLM) which interleaves reasoning and action for segmentation via multi-turn tool invocations.<n> RSAgent queries a segmentation toolbox, observes visual feedback, and revises its spatial hypothesis using historical observations to re-localize targets and iteratively refine masks.<n>Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance
  arXiv  Detail & Related papers  (2025-12-30T06:50:11Z)
• Quantifying Memory Use in Reinforcement Learning with Temporal Range [51.98491034847041]
  Temporal Range is a model-agnostic metric that treats first-order sensitivities of multiple vector outputs across a temporal window to the input sequence as a temporal influence profile.<n>We also report Temporal Range for a compact Long Expressive Memory (LEM) policy trained on the task, using it as a proxy readout of task-level memory.
  arXiv  Detail & Related papers  (2025-12-05T22:58:09Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.