HippoCamp: Benchmarking Contextual Agents on Personal Computers

• URL: http://arxiv.org/abs/2604.01221v1
• Date: Wed, 01 Apr 2026 17:58:33 GMT
• Title: HippoCamp: Benchmarking Contextual Agents on Personal Computers
• Abstract summary: HippoCamp is a new benchmark designed to evaluate agents' capabilities on multimodal file management.<n>Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files.
• Score: 71.97629614361549
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present HippoCamp, a new benchmark designed to evaluate agents' capabilities on multimodal file management. Unlike existing agent benchmarks that focus on tasks like web interaction, tool use, or software automation in generic settings, HippoCamp evaluates agents in user-centric environments to model individual user profiles and search massive personal files for context-aware reasoning. Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files. Building upon the raw files, we construct 581 QA pairs to assess agents' capabilities in search, evidence perception, and multi-step reasoning. To facilitate fine-grained analysis, we provide 46.1K densely annotated structured trajectories for step-wise failure diagnosis. We evaluate a wide range of state-of-the-art multimodal large language models (MLLMs) and agentic methods on HippoCamp. Our comprehensive experiments reveal a significant performance gap: even the most advanced commercial models achieve only 48.3% accuracy in user profiling, struggling particularly with long-horizon retrieval and cross-modal reasoning within dense personal file systems. Furthermore, our step-wise failure diagnosis identifies multimodal perception and evidence grounding as the primary bottlenecks. Ultimately, HippoCamp exposes the critical limitations of current agents in realistic, user-centric environments and provides a robust foundation for developing next-generation personal AI assistants.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information [61.41881137371855]
  We introduce SPIEval, a benchmark for large language models (LLMs)<n> SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools.<n>The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%.
  arXiv  Detail & Related papers  (2026-08-11T09:14:53Z)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks [43.36285223987099]
  We introduce UniClawBench, the first capability-driven benchmark to evaluate proactive agents in dynamic, real-world settings.<n>UniClawBench is built around five executor model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.<n>Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints.
  arXiv  Detail & Related papers  (2026-07-09T17:59:32Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks [49.59854479079552]
  MEGA-Bench is an evaluation suite that scales multimodal evaluation to over 500 real-world tasks.<n>We collected 505 tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space.
  arXiv  Detail & Related papers  (2024-10-14T14:42:12Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.