Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen

Abstract

Event cameras, also known as neuromorphic cameras, havegained significant attention in recent years due to their high tempo-ral resolution, high dynamic range, and low power consumption. Whilemany studies and datasets in neuromorphic vision have focused on auto-motive and drone applications, human-centric daily-life scenarios remainlargely underrepresented, despite their importance for developing andbenchmarking event-based perception systems. Moreover, the few ex-isting event-based human activity datasets are typically recorded withscripted human actions, limiting their ability to capture natural hu-man behaviors. In this paper, we introduce EventKitchen, a large-scalestereo event camera benchmark dataset of human cooking activities inthe kitchen. EventKitchen is egocentrically collected from 10 participantsin 13 diverse kitchens, where the participants wear a helmet with multiplesensors and naturally perform cooking activities, without any scriptedactions. EventKitchen comprises 5.5 hours of stereo event recordingswith synchronized RGB, depth, and IMU data. We provide human an-notations for 10,762 action segments and 13,482 bounding boxes. Wetrain baseline models on EventKitchen to perform multiple event-basedtasks, including action recognition, object detection, and stereo depthestimation. By capturing natural, real-world human activities, Event-Kitchen establishes a challenging benchmark for neuromorphic visionbeyond autonomous driving. The dataset and toolkit are available athttps://chengmingf.github.io/EventKitchen.github.io/