EvDiff: High Quality Video with an Event Camera

摘要

As neuromorphic sensors, event cameras asynchronously recordchanges in brightness as streams of sparse events with the advantages ofhigh temporal resolution and high dynamic range. Reconstructing inten-sity images from events is a highly ill-posed task due to the inherent am-biguity of absolute brightness. Early methods generally follow an end-to-end regression paradigm, directly mapping events to intensity frames ina deterministic manner. While effective to some extent, these approachesoften yield perceptually inferior results and struggle to scale up in modelcapacity and training data. In this work, we propose EvDiff, an event-based diffusion model that follows a surrogate training framework to pro-duce high-quality videos. To reduce the high computational cost of high-frame-rate video generation, we design an event-based diffusion modelthat performs only a single forward diffusion step, equipped with a tem-porally consistent EvEncoder. Furthermore, our novel Surrogate TrainingFramework eliminates the dependence on paired event–image datasets,allowing the model to leverage large-scale image datasets for higher ca-pacity. The proposed EvDiff is capable of generating high-quality col-orful videos solely from monochromatic event streams. Experiments onreal-world datasets demonstrate that our method strikes a sweet spotbetween fidelity and realism, outperforming existing approaches on bothpixel-level and perceptual metrics. The code will be released publicly.