VisualClaw: A Real-Time, Personalized Agent for the Physical World

• URL: http://arxiv.org/abs/2606.16295v1
• Date: Mon, 15 Jun 2026 06:58:22 GMT
• Title: VisualClaw: A Real-Time, Personalized Agent for the Physical World
• Abstract summary: We present VisualClaw, a self-evolving multimodal agent built around two principles.<n>First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate.<n>Second, skill evolution lets the agent learn from failures, producing skill-bank updates that help future questions.
• Score: 76.89211120690028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision language models are serving as general-purpose interfaces for complex multimodal tasks. However, deployment still faces three gaps: VLMs typically incur high latency and cost when processing dense video frames and long prompts, the agent scaffold remains static after deployment, and standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces. We present VisualClaw, a self-evolving multimodal agent built around two principles. First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate and compressing the text skill bank through hot/cold top-k injection. Second, skill evolution lets the agent learn from failures: retrieved memories condition an evolver as direct concatenated context or as guided evidence, producing skill-bank updates that help future questions. Across 4 video-QA benchmarks with 2 VLMs, VisualClaw cuts per-question API cost by an average -98% versus full-frame upload and by -25.9% over the offline uniform 8 frame baseline, while boosting accuracy in most settings, e.g., an average +3.85% and a peak +15.80% on EgoSchema with Gemini 3 Flash. To address the gap, we curate VisualClawArena, a 200-scenario multimodal agentic benchmark built through a strict five-stage pipeline; models must use video evidence, documents, dynamic updates, and executable checks inside a workspace. On VisualClawArena, the same framework with computer-use agent backends improves macro accuracy by +2.9% for Codex (GPT-5.5) and +3.2% for Claude Code (Sonnet 4.6) over no-evolution baselines, with a -9.5% cost reduction compared to the uniform-sampled baseline. These properties make VisualClaw a natural fit for edge applications, where the cascade reduces a 1-hour streaming session from ~3,600 API uploads down to only 5-20 calls and the self-evolution makes it a perfect personalized assistant.

Related papers

• LOCI: A Locator-Critic with Refinement Loop [65.51043732421378]
  Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding.<n>We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image.<n>We propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification.
  arXiv  Detail & Related papers  (2026-08-31T15:26:00Z)
• Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode [0.0]
  Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic.<n>This report argues that the most effective response is to co-design the model architecture and the inference together.<n>It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures.
  arXiv  Detail & Related papers  (2026-08-24T21:28:05Z)
• Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model [60.85752511708308]
  We present Mage-VL, an efficient streaming foundation model for real-time multimodal understanding and interaction.<n>At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions.<n>We establish AI4AI data pipelines encompassing prompt-code joint optimization for captioning and AI-driven performance diagnosis.
  arXiv  Detail & Related papers  (2026-07-27T17:59:53Z)
• StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design [8.502778898017556]
  We present StepX-Edge, a on-device UI vision-language model that resolves tension through three co-grained co-grained architecture, training, and deployment.<n>For training, the five-stage StepX-Curriculum framework is designed around our observation of mutual-promotion effects among UI subtasks.<n>For deployment, a module-wise differentiated two-stage PTQ-to-QAT quantization scheme keeps the post-quantization accuracy loss within 1%.
  arXiv  Detail & Related papers  (2026-07-20T14:14:44Z)
• PP-OCRv6: From 1.5M to 34.5M Parameters, Surpassing Billion-Scale VLMs on OCR Tasks [15.973951643113963]
  PP-OCRv6 is a lightweight OCR system that combines architectural innovation with data-centric optimization.<n>On our in-house benchmarks, PP-OCRv6_medium achieves 83.2% recognition accuracy and 86.2% detection Hmean.
  arXiv  Detail & Related papers  (2026-06-11T09:35:16Z)
• ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces [17.202580606345666]
  Large language model (LLM) agents are increasingly deployed to automate productivity tasks.<n>Existing benchmarks rely on simplified environments and fail to capture realistic, multi-service environments.<n>We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings.
  arXiv  Detail & Related papers  (2026-04-06T21:09:06Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild [74.7263562191605]
  Large language model (LLM) agents are increasingly used for complex tasks.<n>Existing methods either store raw trajectories without distilling knowledge, maintain static skill libraries, or require disruptive downtime for retraining.<n>We present MetaClaw, a continual meta-learning framework that jointly evolves a base LLM policy and a library of reusable behavioral skills.
  arXiv  Detail & Related papers  (2026-03-17T22:30:30Z)
• Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory [50.30283773196725]
  Existing approaches rely on key-value caching to accumulate frame-level details over time, but use a limited number of tokens per frame.<n>We propose scaling the token budget to enable more granular-temporal understanding and reasoning.
  arXiv  Detail & Related papers  (2026-02-20T18:59:50Z)
• VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding [9.415923244280542]
  VideoBrain is an end-to-end framework that enables Vision-Language Models to adaptively acquire visual information through learned sampling policies.<n>Our approach features dual complementary agents: a CLIP-based agent for semantic retrieval across the video and a Uniform agent for dense temporal sampling within intervals.
  arXiv  Detail & Related papers  (2026-02-04T00:08:35Z)
• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos [62.29924199978745]
  VideoAgentTrek is a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale.<n>Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels.<n>applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically.
  arXiv  Detail & Related papers  (2025-10-22T11:25:48Z)
• Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration [9.151759069858924]
  We introduce a principled framework that formalizes the model-agnostic long context modeling problem as a compression problem.<n>We propose Graph of Agents (GoA), which dynamically constructs an input-dependent collaboration structure that maximizes this objective.<n>GoA surpasses the 128K context window Llama 3.1 8B on LongBench, showing a dramatic increase in effective context length.
  arXiv  Detail & Related papers  (2025-09-26T04:15:40Z)
• Spatiotemporal Contrastive Video Representation Learning [87.56145031149869]
  We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn visual representations from unlabeled videos. Our representations are learned using a contrasttemporalive loss, where two augmented clips from the same short video are pulled together in the embedding space. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial.
  arXiv  Detail & Related papers  (2020-08-09T19:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.