Guiding the Blind: Generalizing GUI Agents to Unseen Websites via Multimodal Tutorials

Abstract

Despite the remarkable progress of Large Multimodal Models(LMMs), deploying autonomous agents to navigate web Graphical UserInterfaces (GUIs) remains a significant challenge. Most existing agentsare “blind” when encountering unfamiliar websites, as they rely heav-ily on patterns memorized during in-domain training, which inevitablyfails in the open-world web. To bridge this generalization gap, we pro-pose that agents should mimic human behavior: leveraging external ex-pertise to navigate unknown environments. In this paper, we introduceWebOne, a novel benchmark designed to evaluate an agent’s ability to mas-ter unseen websites by referencing Multimodal Tutorials—heterogeneousknowledge sources derived from instructional videos, historical trajec-tories, and human demonstrations. WebOne comprises 1,342 real-worldtasks and 970 high-quality tutorials across 60+ websites. Building uponthis, we propose WebLearner, a reinforcement learning-based frameworkthat utilizes a hierarchical referencing strategy to synthesize informa-tion from these multimodal tutorials. Experimental results demonstratethat WebLearner significantly outperforms current state-of-the-art open-source models, achieving a 56.9% success rate on held-out websites andshowing competitive performance against recent proprietary models.