GameWorlds: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
Authors: Mingyu Ouyang, Siyuan Hu, Qinghong Lin, Hwee Tou Ng, Mike Zheng Shou
Abstract
Towards an embodied generalist for real-world interaction,Multimodal Large Language Model (MLLM) agents still suffer fromchallenging latency, sparse feedback, and irreversible mistakes. Videogames offer an ideal testbed with rich visual observations and closed-loopinteraction, demanding fine-grained perception, long-horizon planning,and precise control. However, systematically evaluating these capabilitiesis currently hindered by heterogeneous action interfaces and heuristicverification. To this end, we introduce GameWorld, a benchmarkdesigned for standardized and verifiable evaluation of MLLMs as generalistgame agents in browser environments. Two game agent interfaces arestudied: (i) Computer-use agents that directly emit keyboard and mousecontrols, and (ii) Generalist multimodal agents that act in a semanticaction space via deterministic Semantic Action Parsing. GameWorldcontains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18model-interface pairs suggest that even the best-performing agent isfar from achieving human capabilities on video games. Further studieson real-time interaction, context-memory sensitivity, and action validityexpose more challenges ahead for game agents. Together, by offering astandardized and verifiable evaluation framework, GameWorld laysa foundation for advancing research on multimodal game agents andbeyond.