Nexels: Neurally-Textured Surfels for Real-Time Novel View Synthesis with Sparse Primitives

Abstract

Though Gaussian splatting has achieved impressive resultsin novel view synthesis, it requires millions of primitives to representhighly textured scenes — even when the underlying geometry is simple.We propose a representation that decouples geometry and appearanceto achieve a compact and computationally efficient representation. Ourmethod uses surfels for geometry and combines a global neural field withper-primitive colors for appearance. While existing approaches that tex-ture primitives incur significant overhead in either memory or renderingspeed, our method strikes a better balance by leveraging the neural fieldto implicitly compress texture information. We bound the computationsrequired in the rendering pass by querying the neural field for a fixednumber of primitives per pixel, enabling real-time rendering speeds. Ourrepresentation matches the perceptual quality of 3D Gaussian splattingwhile using 9.7× fewer primitives and 5.5× less memory on outdoorscenes, and 31× fewer primitives and 3.7× less memory on indoor scenes.