SIGGRAPH 2020 Papers

Due to the limited range of depth sensors, the depth estimators for wild images were trained by ordinal data. Without the need of re-train, we proposed an approach to create seamlessly ordinal depth for 360 wild images. The 360 depth was applied to VR display for improving the virtual perception.

Seamless global parametrization of surfaces is a key operation in geometry processing, e.g., for high-quality quad mesh generation. A common approach is to prescribe the parametric domain structure, in particular, the locations of parametrization singularities (cones), and solve a non-convex optimization problem minimizing a distortion measure, with local injectivity imposed through either constraints or barrier terms. In both cases, an initial valid parametrization is essential to serve as a feasible starting point for obtaining an optimized solution. While convexified versions of the constraints eliminate this initialization requirement, they narrow the range of solutions, causing some problem instances that actually do have a solution to become infeasible. We demonstrate that for arbitrary given sets of topologically admissible parametric cones with prescribed curvature, a global seamless parametrization always exists (with the exception of one well-known case). Importantly, our proof is constructive and directly leads to a general algorithm for computing such parametrizations. Most distinctively, this algorithm is bootstrapped with a convex optimization problem (solving for a conformal map), in tandem with a simple linear equation system (determining a seamless modification of this map). This initial map can then serve as a valid starting point and be optimized for low distortion using existing injectivity preserving methods.

A powerful rigging pipeline is proposed to automatically rig the raw scans of human model with 1) fitting a Rigged Parametric Body Model (RPBM) to the scan with a novel and effective energy formulation, 2) cutting the scan to break self-occlusions automatically, 3) inpainting the geometry and texture in UV parametric space to create a complete avatar and 4) transferring the rigging information from the RPBM to the avatar. To further improve the fitting result, Semantic Deformation Component (SDC) is generated and utilized to replace the original shape blendshapes of the RPBM in the fitting stage, which can also be used for body reshaping. Our method is highly effective and robust in rigging the raw human scan models.

In this work we present a novel, robust transition generation technique that can serve as a new tool for 3D animators, based on adversarial recurrent neural networks. The system synthesises high-quality motions that use temporally-sparse keyframes as animation constraints. This is reminiscent of the job of in-betweening in traditional animation pipelines, in which an animator draws motion frames between provided keyframes. We first show that a state-of-the-art motion prediction model cannot be easily converted into a robust transition generator when only adding conditioning information about future keyframes. To solve this problem, we then propose two novel additive embedding modifiers that are applied at each timestep to latent representations encoded inside the network's architecture. One modifier is a time-to-arrival embedding that allows variations of the transition length with a single model. The other is a scheduled target noise vector that allows the system to be robust to target distortions and to sample different transitions given fixed keyframes. To qualitatively evaluate our method, we present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios. To quantitatively evaluate performance on transitions and generalizations to longer time horizons, we present well-defined in-betweening benchmarks on a subset of the widely used Human3.6M dataset and on LaFAN1, a novel high quality motion capture dataset that is more appropriate for transition generation. We are releasing this new dataset along with this work, with accompanying code for reproducing our baseline results.

Effective local light transport guiding demands for high quality guiding information, i.e., a precise representation of the directional incident radiance distribution at every point inside the scene. We introduce a parallax-aware distribution model based on parametric mixtures. By parallax-aware warping of the distribution, the local approximation of the 5D radiance field remains valid and precise across large spatial regions, even for close-by contributors. Our robust optimization scheme fits parametric mixtures to radiance samples collected in previous rendering passes. Robustness is achieved by splitting and merging of components refining the mixture. These splitting and merging decisions minimize and bound the expected variance of the local radiance estimator. In addition, we extend the fitting scheme to a robust, iterative update method, which allows for incremental training of our model using smaller sample batches. This results in more frequent training updates and, at the same time, significantly reduces the required sample memory footprint. The parametric representation of our model allows for the application of advanced importance sampling methods such as radiance-based, cosine-aware, and even product importance sampling. Our method further smoothly integrates next-event estimation (NEE) into path guiding, avoiding importance sampling of contributions better covered by NEE. The proposed robust fitting and update scheme, in combination with the parallax-aware representation, results in faster learning and lower variance compared to state-of-the-art path guiding approaches.

This paper introduces a method to simulate complex rod assemblies and stacked layers with implicit contact handling, through Eulerian-on-Lagrangian (EoL) discretizations. Previous EoL methods fail to handle such complex situations, due to ubiquitous and intrinsic degeneracies in the contact geometry, which prevent the use of remeshing and make simulations unstable. We propose a novel mixed Eulerian-Lagrangian discretization that supports accurate and efficient contact as in EoL methods, but is transparent to internal rod forces, and hence insensitive to degeneracies. By combining the standard and novel EoL discretizations as appropriate, we derive mixed statics-dynamics equations of motion that can be solved in a unified manner with standard solvers. Our solution is simple and elegant in practice, and produces robust simulations on large-scale scenarios with complex rod arrangements and pervasive degeneracies. We demonstrate our method on multi-layer yarn-level cloth simulations, with implicit handling of both intra-and inter-layer contacts.

Hot-wire cutting is a subtractive fabrication technique used to carve foam and similar materials. Conventional machines rely on straight wires and are thus limited to creating piecewise ruled surfaces. In this work, we propose a method that exploits a dual-arm robot setup to actively control the shape of a flexible, heated rod as it cuts through the material. While this setting offers great freedom of shape, using it effectively requires concurrent reasoning about three tightly coupled sub-problems: 1) modeling the way in which the shape of the rod and the surface it sweeps are governed by the robot's motions; 2) approximating a target shape through a sequence of surfaces swept by the equilibrium shape of an elastic rod; and 3) generating collision-free motion trajectories that lead the robot to create desired sweeps with the deformable tool. We present a computational framework for robotic hot wire cutting that addresses all three sub-problems in a unified manner. We evaluate our approach on a set of simulated results and physical artefacts generated with our robotic fabrication system.

We present RigNet, an end-to-end automated method for producing animation rigs from input character models. Given an input 3D model representing an articulated character, RigNet predicts a skeleton that matches the animator expectations in joint placement and topology. It also estimates surface skin weights based on the predicted skeleton. Our method is based on a deep architecture that directly operates on the mesh representation without making assumptions on shape class and structure. The architecture is trained on a large and diverse collection of rigged models, including their mesh, skeletons and corresponding skin weights. Our evaluation is three-fold: we show better results than prior art when quantitatively compared to animator rigs; qualitatively we show that our rigs can be expressively posed and animated at multiple levels of detail; and finally, we evaluate the impact of various algorithm choices on our output rigs.1

Bidirectional path tracing (BPT) with multiple importance sampling (MIS) is a popular technique for rendering realistic images. Recently, it has been shown that BPT can be improved by preparing multiple light sub-paths and by resampling a small number of light sub-paths from them to generate full paths with large contribution. Traditionally, for MIS weights, the balance heuristic has widely been used to minimize the upper bound of variance, where each full path is weighted in proportion to the probability of the path. Although the probability of the path can change due to the resampling process, the weighting functions used in the previous methods remain unaffected by the change in probability, resulting in less efficiency. To address this problem, we propose new weighting functions for BPT with multiple light sub-paths. Our main contribution is a precise formulation of the variance and the derivation of the weighting functions that can appropriately treat the change in probability. We demonstrate that our weighting functions significantly improve the image quality. We will release a simple version of our implementation as open source to ensure reproducibility.

Crochet is a fabrication technique in which a 3D surface is created from yarn by interlacing loops formed with a special hook. Crochet patterns are typically represented using a standardized set of abstract pictorial symbols. Unfortunately, while this notation is enough for someone well-versed in the individual stitches, it does not directly show the yarn layout of stitches. This lack of specification makes it difficult for both novice users and computer programs to parse, visualize, and design crochet patterns. We demonstrate how to represent crochet patterns within the “stitch mesh” paradigm. That is, the pattern is represented using a library of tiles, where each tile contains yarn geometry, and tiles connect along their edges. In order to adapt stitch meshes to crochet, we introduce a special edge type which captures the idea of the current loop – the loop of yarn held on the crochet hook during fabrication. We also create a library of mesh face types which model commonly-used crochet stitches. We illustrate the richness of the crochet stitch faces by showing a number of examples including patterns generated from 3D models.

We present a rendering system for 4D ultrasound data based on Monte Carlo path tracing, where a recurrent denoising autoencoder is trained on a large collection of images to produce noise-free images with a reduced number of samples per pixel. While the diagnostic value of photorealistic shading for 3D medical imaging has not been established definitively, the enhanced shape and depth perception allow for a more complete understanding of the data in a variety of scenarios. The dynamic nature of ultrasound data typically limits the global illumination effects that can be rendered interactively, but we demonstrated that AI-based denoising together with Monte Carlo path tracing can be used both for interactive workflows and for rendering an entire heartbeat sequence at high quality in about a minute, while also allowing for complex lighting environments. Specifically, our contribution is a model compatible with the NVIDIA OptiX interactive denoiser, which has been trained on ultrasound-specific rendering presets and data.

In this paper, we present the concept of Remote Empathetic Viewpoints, an approach which allows the viewer to simultaneously access multiple remote viewpoints, granting us the ability to explore and extend the concepts of Cubism in 3D and in animation. In Remote Empathetic Viewpoints, we utilize a single Primary View Camera, a small set of Control lights, and a significantly large number of secondary cameras, whose positions and directions are referenced by Control Lights. By using such a multi camera system, we overcome the “shower door” affect that comes from using multiple cameras, which are used to obtain cubist rendering. By animating camera movement, we obtain temporal cubist art.

We introduce an efficient method for designing shell reinforcements of minimal weight. Inspired by classical Michell trusses, we create a reinforcement layout whose members are aligned with optimal stress directions, then optimize their shape minimizing the volume while keeping stresses bounded. We exploit two predominant techniques for reinforcing shells: adding ribs aligned with stress directions and using thicker walls on regions of high stress. Most previous work can generate either only ribs or only variable-thickness walls. However, in the general case, neither approach by itself will provide optimal solutions. By using a more precise volume model, our method is capable of producing optimized structures with the full range of qualitative behaviors: from ribs to walls and smoothly transitioning in between. Our method includes new algorithms for determining the layout of reinforcement structure elements, and an efficient algorithm to optimize their shape, minimizing a non-linear non-convex functional at a fraction of the cost and with better optimality compared to standard solvers. We demonstrate the optimization results for a variety of shapes and the improvements it yields in the strength of 3D-printed objects.

We present a valid polarization-based reflection contaminated image synthesis method, which can provide adequate, diverse and authentic training dataset. Meanwhile, we enhance the neural network by introducing the reflection information as guidance and utilizing adaptive convolution kernel size to fuse multi-scale information. We demonstrate that the proposed approach achieves convincing improvements over state of the arts.

Edge-preserving image smoothing is a fundamental procedure for many computer vision and graphic applications. There is a tradeoff between the smoothing quality and the processing speed: the high smoothing quality usually requires a high computational cost, which leads to the low processing speed. In this article, we propose a new global optimization based method, named iterative least squares (ILS), for efficient edge-preserving image smoothing. Our approach can produce high-quality results but at a much lower computational cost. Comprehensive experiments demonstrate that the proposed method can produce results with little visible artifacts. Moreover, the computation of ILS can be highly parallel, which can be easily accelerated through either multi-thread computing or the GPU hardware. With the acceleration of a GTX 1080 GPU, it is able to process images of 1080p resolution (1920 × 1080) at the rate of 20fps for color images and 47fps for gray images. In addition, the ILS is flexible and can be modified to handle more applications that require different smoothing properties. Experimental results of several applications show the effectiveness and efficiency of the proposed method. The code is available at https://github.com/wliusjtu/Real-time-Image-Smoothing-via-Iterative-Least-Squares.

In this research, we propose a digital fabrication method that can print reusable objects rapidly using the clay material. Therefore, we developed a magnetic force control device and a magnetic thermoplastic material that deforms dynamically. When the user creates a shape model on the screen, the clay material instantly changes to the shape of the designed model. The user can confirm the shape in a short time, and the material can be reused. In this paper, we describe the design and implementation of magnetic materials and control devices, evaluation, and future work.

Physically based differentiable rendering has recently evolved into a powerful tool for solving inverse problems involving light. Methods in this area perform a differentiable simulation of the physical process of light transport and scattering to estimate partial derivatives relating scene parameters to pixels in the rendered image. Together with gradient-based optimization, such algorithms have interesting applications in diverse disciplines, e.g., to improve the reconstruction of 3D scenes, while accounting for interreflection and transparency, or to design meta-materials with specified optical properties. The most versatile differentiable rendering algorithms rely on reverse-mode differentiation to compute all requested derivatives at once, enabling optimization of scene descriptions with millions of free parameters. However, a severe limitation of the reverse-mode approach is that it requires a detailed transcript of the computation that is subsequently replayed to back-propagate derivatives to the scene parameters. The transcript of typical renderings is extremely large, exceeding the available system memory by many orders of magnitude, hence current methods are limited to simple scenes rendered at low resolutions and sample counts. We introduce radiative backpropagation, a fundamentally different approach to differentiable rendering that does not require a transcript, greatly improving its scalability and efficiency. Our main insight is that reverse-mode propagation through a rendering algorithm can be interpreted as the solution of a continuous transport problem involving the partial derivative of radiance with respect to the optimization objective. This quantity is "emitted" by sensors, "scattered" by the scene, and eventually "received" by objects with differentiable parameters. Differentiable rendering then decomposes into two separate primal and adjoint simulation steps that scale to complex scenes rendered at high resolutions. We also investigated biased variants of this algorithm and find that they considerably improve both runtime and convergence speed. We showcase an efficient GPU implementation of radiative backpropagation and compare its performance and the quality of its gradients to prior work.

Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons, and capturing their time-of-arrival with high timing precision. While these sensors were limited to singlepixel or low-resolution devices in the past, recently, large (up to 1 MPixel) SPAD arrays have been developed. These single-photon cameras (SPCs) are capable of capturing high-speed sequences of binary single-photon images with no read noise. We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions, including ultra low-light and fast motion. Inspired by recent success of conventional burst photography, we design algorithms that align and merge binary sequences captured by SPCs into intensity images with minimal motion blur and artifacts, high signal-to-noise ratio (SNR), and high dynamic range. We theoretically analyze the SNR and dynamic range of quanta burst photography, and identify the imaging regimes where it provides significant benefits. We demonstrate, via a recently developed SPAD array, that the proposed method is able to generate high-quality images for scenes with challenging lighting, complex geometries, high dynamic range and moving objects. With the ongoing development of SPAD arrays, we envision quanta burst photography finding applications in both consumer and scientific photography.