HippoCamp: Benchmarking Contextual Agents on Personal Computers

摘要

We present HippoCamp, a new benchmark designed to eval-uate agents’ capabilities on multimodal file management. Unlike exist-ing agent benchmarks that focus on tasks like web interaction, tool-use, or software automation in generic settings, HippoCamp evaluatesagents in user-centric environments to model individual user profilesand search from massive personal files for context-aware reasoning. Ourbenchmark instantiates device-scale file systems over real-world profilesspanning diverse modalities, comprising 42.4 GB of data across over2K heterogeneous files. Building upon the raw files, we construct 581QA pairs to assess agents’ capabilities in search, evidence perception,and multi-step reasoning. To facilitate fine-grained analysis, we pro-vide 46.1K fine-grained trajectory annotations for step-wise failure di-agnosis. We evaluate a wide range of state-of-the-art multimodal large✉ Corresponding authors.language models (MLLMs) and agentic methods on HippoCamp. Ourcomprehensive experiments reveal a significant performance gap: eventhe most advanced commercial models achieve merely a 48.3% accu-racy in user profiling, struggling particularly with long-horizon retrievaland cross-modal reasoning within dense personal file systems. Further-more, our step-wise failure diagnosis identifies multimodal perceptionand evidence grounding as the primary bottlenecks. Ultimately, Hip-poCamp exposes the critical limitations of current agents in realistic,user-centric environments and provides a robust foundation for develop-ing next-generation personal AI assistants. Our dataset is publicly avail-able at: https://huggingface.co/datasets/MMMem-org/HippoCamp.