WildCity: A Real-World Dataset for City-Scale Rendering and Beyond

Abstract

Humans can navigate an unfamiliar city and gradually forma coherent spatial mental map spanning tens of square kilometers. CanAI build spatial representations at a comparable scale? Although recentfoundation models have advanced scene reconstruction and embodied in-telligence, scaling to entire cities remains an open challenge, primarilydue to the lack of city-scale data. To bridge the gap, we introduce Wild-City, a real-world multimodal dataset collected by autonomous fleetstraversing complex urban environments. Our dataset includes 18 tra-jectories, each averaging 83.7 kilometers in length, and preserves thecore challenges of in-the-wild perception, e.g., dynamic objects, lightingvariations, and imperfect camera poses. We further establish an urban-tailored reconstruction baseline and convert the reconstructed environ-ments into a closed-loop simulator. Beyond the dataset and baseline, wesystematically analyze the key challenges on the path to simulation-readyurban digital twins: scalability, extrapolation, and uncertainty. Ul-timately, WildCity aims to catalyze progress not only in city-scale render-ing, but more broadly in the pursuit of AI that can perceive, remember,and reason across space at a scale comparable to human cognition.