SIGGRAPH Asia 2023 Papers

We introduce an adaptive level-of-detail technique for ray tracing triangle meshes that aims to reduce the memory bandwidth used during ray traversal, which can be the bottleneck for rendering time with large scenes and the primary consumer of energy. We propose a specific data structure for hierarchically representing triangle meshes, allowing localized decisions for the desired mesh resolution per ray. Starting with the lowest-resolution triangle mesh level, higher-resolution levels are generated by tessellating each triangle into four via splitting its edges with arbitrarily-placed vertices. We fit the resulting mesh hierarchy into a specialized acceleration structure to perform on-the-fly tessellation level selection during ray traversal. Our structure reduces both storage cost and data movement during rendering, which are the main consumers of energy. It also allows continuous transitions between detail levels, while locally adjusting the mesh resolution per ray and preserving watertightness. We present how this structure can be used with both primary and secondary rays for reflections and shadows, which can intersect with different tessellation levels, providing consistent results. We also propose specific hardware units to cover the cost of additional compute needed for level-of-detail operations. We evaluate our method using a cycle-accurate simulation of a custom ray tracing hardware architecture. Our results show that, as compared to traditional bounding volume hierarchies, our method can provide more than an order of magnitude reduction in energy use and render time, given sufficient computational resources.

Existing real-time RGB-D reconstruction approaches, like Kinect Fusion, lack real-time photo-realistic visualization. This is due to noisy, oversmoothed or incomplete geometry and blurry textures which are fused from imperfect depth maps and camera poses. Recent neural rendering methods can overcome many of such artifacts but are mostly optimized for offline usage, hindering the integration into a live reconstruction pipeline. In this paper, we present LiveNVS, a system that allows for neural novel view synthesis on a live RGB-D input stream with very low latency and real-time rendering. Based on the RGB-D input stream, novel views are rendered by projecting neural features into the target view via a densely fused depth map and aggregating the features in image-space to a target feature map. A generalizable neural network then translates the target feature map into a high-quality RGB image. LiveNVS achieves state-of-the-art neural rendering quality of unknown scenes during capturing, allowing users to virtually explore the scene and assess reconstruction quality in real-time.

Kirchhoff-Love shells are commonly used in many branches of engineering, including in computer graphics, but have so far been simulated only under limited nonlinear material options. We derive the Kirchhoff-Love thin-shell mechanical energy for an arbitrary 3D volumetric hyperelastic material, including isotropic materials, anisotropic materials, and materials whereby the energy includes both even and odd powers of the principal stretches. We do this by starting with any 3D hyperelastic material, and then analytically computing the corresponding thin-shell energy limit. This explicitly identifies and separates in-plane stretching and bending terms, and avoids numerical quadrature. Thus, in-plane stretching and bending are shown to originate from one and the same process (volumetric elasticity of thin objects), as opposed to from two separate processes as done traditionally in cloth simulation. Because we can simulate materials that include both even and odd powers of stretches, we can accommodate standard mesh distortion energies previously employed for 3D solid simulations, such as Symmetric ARAP and Co-rotational materials. We relate the terms of our energy to those of prior work on Kirchhoff-Love thin-shells in computer graphics that assumed small in-plane stretches, and demonstrate the visual difference due to the presence of our exact stretching and bending terms. Furthermore, our formulation allows us to categorize all distinct hyperelastic Kirchhoff-Love thin-shell energies. Specifically, we prove that for Kirchhoff-Love thin-shells, the space of all hyperelastic materials collapses to two-dimensional hyperelastic materials. This observation enables us to create an interface for the design of thin-shell Kirchhoff-Love mechanical energies, which in turn enables us to create thin-shell materials that exhibit arbitrary stiffness profiles under large deformations.

K-surfaces are an interactive modeling technique for Bézier-spline surfaces. Inspired by k-curves by [Yan et al. 2017], each patch provides a single control point that is being interpolated at a local extremum of Gaussian curvature. The challenge is to solve the inverse problem of finding the center control point of a Bézier patch given the boundary control points and the handle. Unlike the situation in 2D, bi-quadratic Bézier patches may exhibit none, one, or several extrema, and finding them is non-trivial. We solve the difficult inverse problem, including the possible selection among several extrema, by learning the desired function from samples, generated by computing Gaussian curvature of random patches. This approximation provides a stable solution to the ill-defined inverse problem and is much more efficient than direct numerical optimization, facilitating the interactive modeling framework. The local solution is used in an iterative optimization incorporating continuity constraints across patches. We demonstrate that the surface varies smoothly with the handle location and that the resulting modeling system provides local and generally intuitive control. The idea of learning the inverse mapping from handles to patches may be applicable to other parametric surfaces.

When dealing with difficult inverse problems such as inverse rendering, using Monte Carlo estimated gradients to optimise parameters can slow down convergence due to variance. Averaging many gradient samples in each iteration reduces this variance trivially. However, for problems that require thousands of optimisation iterations, the computational cost of this approach rises quickly. We derive a theoretical framework for interleaving sampling and optimisation. We update and reuse past samples with low-variance finite-difference estimators that describe the change in the estimated gradients between each iteration. By combining proportional and finite-difference samples, we continuously reduce the variance of our novel gradient meta-estimators throughout the optimisation process. We investigate how our estimator interlinks with Adam and derive a stable combination. We implement our method for inverse path tracing and demonstrate how our estimator speeds up convergence on difficult optimisation tasks.

Image-space denoising techniques have been widely employed in Monte Carlo rendering, typically blending neighboring pixel estimates using a denoising kernel. It is widely recognized that a kernel should be adapted to characteristics of the input pixel estimates in order to ensure robustness to diverse image features and amount of noise. Denoising with such an input-dependent kernel, however, can introduce a bias that makes the denoised estimate even less accurate than the noisy input estimate. Consequently, it has been considered essential to balance the bias introduced by denoising and the reduction of noise. We propose a new framework to define an input-dependent kernel that departs from the existing approaches based on error estimation or supervised learning. Rather than seeking an optimal bias-noise balance as in those existing approaches, we propose to constrain the amount of bias introduced by denoising. Such a constraint is made possible by the concept of uncorrelated statistics, which has never been applied for denoising. By designing an input-dependent kernel with uncorrelated weights against the input pixel estimates, our denoising kernel can reduce data-dependent noise with a negligible amount of bias in most cases. We demonstrate the effectiveness of our method for various scenes.

Novel-view synthesis is an ill-posed problem in that it requires inference of previously unseen information. Recently, reviving the traditional field of image-based rendering, neural methods proved particularly suitable for this interpolation/extrapolation task; however, they often require a-priori scene-completeness or costly preprocessing steps and generally suffer from long (scene-specific) training times. Our work draws from recent progress in neural spatio-temporal supersampling to enhance a state-of-the-art neural renderer’s ability to infer novel-view information at inference time. We adapt a supersampling architecture [Xiao et al. 2020], which resamples previously rendered frames, to instead recombine nearby camera images in a multi-view dataset. These input frames are warped into a joint target frame, guided by the most recent (point-based) scene representation, followed by neural interpolation. The resulting architecture gains sufficient robustness to significantly improve transferability to previously unseen datasets. In particular, this enables novel applications for neural rendering where dynamically streamed content is directly incorporated in a (neural) image-based reconstruction of a scene. As we will show, our method reaches state-of-the-art performance when compared to previous works that rely on static and sufficiently densely sampled scenes; in addition, we demonstrate our system’s particular suitability for dynamically streamed content, where our approach is able to produce high-fidelity novel-view synthesis even with significantly fewer available frames than competing neural methods.