SIGGRAPH 2021 论文列表

Portraiture as an art form has evolved from realistic depiction into a plethora of creative styles. While substantial progress has been made in automated stylization, generating high quality stylistic portraits is still a challenge, and even the recent popular Toonify suffers from several artifacts when used on real input images. Such StyleGAN-based methods have focused on finding the best latent inversion mapping for reconstructing input images; however, our key insight is that this does not lead to good generalization to different portrait styles. Hence we propose AgileGAN, a framework that can generate high quality stylistic portraits via inversion-consistent transfer learning. We introduce a novel hierarchical variational autoencoder to ensure the inverse mapped distribution conforms to the original latent Gaussian distribution, while augmenting the original space to a multi-resolution latent space so as to better encode different levels of detail. To better capture attribute-dependent stylization of facial features, we also present an attribute-aware generator and adopt an early stopping strategy to avoid overfitting small training datasets. Our approach provides greater agility in creating high quality and high resolution (1024×1024) portrait stylization models, requiring only a limited number of style exemplars (~100) and short training time (~1 hour). We collected several style datasets for evaluation including 3D cartoons, comics, oil paintings and celebrities. We show that we can achieve superior portrait stylization quality to previous state-of-the-art methods, with comparisons done qualitatively, quantitatively and through a perceptual user study. We also demonstrate two applications of our method, image editing and motion retargeting.

In this research, we propose an adaptive radiometric compensation method, which uses only a projector and a camera, on continuously deforming projection surfaces. Radiometric compensation has been widely studied as a technique for making various objects usable as screens, by canceling out the influence of the color and pattern of the projection target. However, since it is necessary to continuously maintain the inter-pixel correspondence between a projector and a camera, to date, the projection target has been limited to stationary objects. Therefore, we propose a method to estimate the inter-pixel correspondence in real-time, using only a normal projector and camera. The method expands the scope of application of projection mapping greatly, by applying radiometric compensation to deforming clothes, and making them available as screens.

Synthesizing graceful and life-like behaviors for physically simulated characters has been a fundamental challenge in computer animation. Data-driven methods that leverage motion tracking are a prominent class of techniques for producing high fidelity motions for a wide range of behaviors. However, the effectiveness of these tracking-based methods often hinges on carefully designed objective functions, and when applied to large and diverse motion datasets, these methods require significant additional machinery to select the appropriate motion for the character to track in a given scenario. In this work, we propose to obviate the need to manually design imitation objectives and mechanisms for motion selection by utilizing a fully automated approach based on adversarial imitation learning. High-level task objectives that the character should perform can be specified by relatively simple reward functions, while the low-level style of the character's behaviors can be specified by a dataset of unstructured motion clips, without any explicit clip selection or sequencing. For example, a character traversing an obstacle course might utilize a task-reward that only considers forward progress, while the dataset contains clips of relevant behaviors such as running, jumping, and rolling. These motion clips are used to train an adversarial motion prior, which specifies style-rewards for training the character through reinforcement learning (RL). The adversarial RL procedure automatically selects which motion to perform, dynamically interpolating and generalizing from the dataset. Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips. Composition of disparate skills emerges automatically from the motion prior, without requiring a high-level motion planner or other task-specific annotations of the motion clips. We demonstrate the effectiveness of our framework on a diverse cast of complex simulated characters and a challenging suite of motor control tasks.

Neural representations have emerged as a new paradigm for applications in rendering, imaging, geometric modeling, and simulation. Compared to traditional representations such as meshes, point clouds, or volumes they can be flexibly incorporated into differentiable learning-based pipelines. While recent improvements to neural representations now make it possible to represent signals with fine details at moderate resolutions (e.g., for images and 3D shapes), adequately representing large-scale or complex scenes has proven a challenge. Current neural representations fail to accurately represent images at resolutions greater than a megapixel or 3D scenes with more than a few hundred thousand polygons. Here, we introduce a new hybrid implicit-explicit network architecture and training strategy that adaptively allocates resources during training and inference based on the local complexity of a signal of interest. Our approach uses a multiscale block-coordinate decomposition, similar to a quadtree or octree, that is optimized during training. The network architecture operates in two stages: using the bulk of the network parameters, a coordinate encoder generates a feature grid in a single forward pass. Then, hundreds or thousands of samples within each block can be efficiently evaluated using a lightweight feature decoder. With this hybrid implicit-explicit network architecture, we demonstrate the first experiments that fit gigapixel images to nearly 40 dB peak signal-to-noise ratio. Notably this represents an increase in scale of over 1000X compared to the resolution of previously demonstrated image-fitting experiments. Moreover, our approach is able to represent 3D shapes significantly faster and better than previous techniques; it reduces training times from days to hours or minutes and memory requirements by over an order of magnitude.

We assume that the viscous forces in any liquid are simultaneously local and non-local, and introduce the extended POM-POM model [McLeish and Larson 1998; Oishi et al. 2012; Verbeeten et al. 2001] to computer graphics to design a unified constitutive model for viscosity that generalizes prior models, such as Oldroyd-B, the Upper-convected Maxwell (UCM) model [Sadeghy et al. 2005], and classical Newtonian viscosity under one umbrella, recovering each of them with different parameter values. Implicit discretization of our model via backward Euler recovers the variational Stokes solver of [Larionov et al. 2017] for Newtonian viscosity. For greater accuracy, however, we introduce the second-order accurate Generalized Single Step Single Solve (GS4) scheme [Tamma et al. 2000; Zhou and Tamma 2004] to computer graphics, which recovers all prior second-order accurate time integration schemes to date. Using GS4 and our generalized constitutive model, we present a Material Point Method (MPM) for simulating various viscoelastic liquid behaviors, such as classical liquid rope coiling, buckling, folding, and shear thinning/thickening. In addition, we show how to couple our viscoelastic liquid simulator with the recently introduced non-Fourier heat diffusion solver [Xue et al. 2020] for simulating problems with phase change, such as melting chocolate and digital fabrication with 3D printing. While the discretization of heat diffusion is slightly different within GS4, we show that it can still be efficiently solved using an assembly-free Multigrid-preconditioned Conjugate Gradients solver. We present end-to-end 3D simulations to demonstrate the versatility of our framework.

We present a purely geometric, time-independent deformer resolving local contacts between elastic objects, including self-collisions between adjacent parts of the same object that often occur in character skinning animation. Starting from multiple meshes in intersection, our deformer first computes the parts of the surfaces remaining in contact, and then applies a procedural displacement with volume preservation. Although our deformer processes each frame independently, it achieves temporally continuous deformations with artistic control of the bulge through few pseudo-stiffness parameters. The plausibility of the deformation is further enhanced by anisotropically spreading the volume-preserving bulge. The result is a robust, real-time deformer that can handle complex geometric configurations such as a ball squashed by a hand, colliding lips, bending fingers, etc.

Cloth dynamics and collision handling are the two most challenging topics in cloth simulation. While researchers have substantially improved the performances of cloth dynamics solvers recently, their success in fast collision detection and handling is rather limited. In this article, we focus our research on the safety, efficiency, and realism of the repulsion-based collision handling approach, which has demonstrated its potential in existing GPU-based simulators. Our first discovery is the necessary vertex distance conditions for cloth to enter self intersections, the negations of which can be viewed as vertex distance constraints continuous in time for sufficiently avoiding self collisions. Continuous constraints, however, cannot be enforced with ease. Our solution is to convert continuous constraints into three types of constraints: discrete edge length constraints, discrete vertex distance constraints, and vertex displacement constraints. Based on this solution, we develop a fast and safe collision handling process for enforcing constraints, a novel splitting method for integrating collision handling with dynamics solvers, and static and adaptive remeshing schemes to further improve the runtime performance. In summary, our cloth simulator is efficient, safe, robust, and parallelizable on a GPU. The experiment shows that it runs at least one order of magnitude faster than existing simulators.

Virtual and augmented reality (VR/AR) displays strive to provide a resolution, framerate and field of view that matches the perceptual capabilities of the human visual system, all while constrained by limited compute budgets and transmission bandwidths of wearable computing systems. Foveated graphics techniques have emerged that could achieve these goals by exploiting the falloff of spatial acuity in the periphery of the visual field. However, considerably less attention has been given to temporal aspects of human vision, which also vary across the retina. This is in part due to limitations of current eccentricity-dependent models of the visual system. We introduce a new model, experimentally measuring and computationally fitting eccentricity-dependent critical flicker fusion thresholds jointly for both space and time. In this way, our model is unique in enabling the prediction of temporal information that is imperceptible for a certain spatial frequency, eccentricity, and range of luminance levels. We validate our model with an image quality user study, and use it to predict potential bandwidth savings 7X higher than those afforded by current spatial-only foveated models. As such, this work forms the enabling foundation for new temporally foveated graphics techniques.

We introduce a novel transmittance model to improve the volumetric representation of 3D scenes. The model can represent opaque surfaces in the volumetric light transport framework. Volumetric representations are useful for complex scenes, and become increasingly popular for level of detail and scene reconstruction. The traditional exponential transmittance model found in volumetric light transport cannot capture correlations in visibility across volume elements. When representing opaque surfaces as volumetric density, this leads to both bloating of silhouettes and light leaking artifacts. By introducing a parametric non-exponential transmittance model, we are able to approximate these correlation effects and significantly improve the accuracy of volumetric appearance representation of opaque scenes. Our parametric transmittance model can represent a continuum between the linear transmittance that opaque surfaces exhibit and the traditional exponential transmittance encountered in participating media and unstructured geometries. This covers a large part of the spectrum of geometric structures encountered in complex scenes. In order to handle the spatially varying transmittance correlation effects, we further extend the theory of non-exponential participating media to a heterogeneous transmittance model. Our model is compact in storage and computationally efficient both for evaluation and for reverse-mode gradient computation. Applying our model to optimization algorithms yields significant improvements in volumetric scene appearance quality. We further show improvements for relevant applications, such as scene appearance prefiltering, image-based scene reconstruction using differentiable rendering, neural representations, and compare it to a conventional exponential model.

We present a novel Material Point Method (MPM) discretization of surface tension forces that arise from spatially varying surface energies. These variations typically arise from surface energy dependence on temperature and/or concentration. Furthermore, since the surface energy is an interfacial property depending on the types of materials on either side of an interface, spatial variation is required for modeling the contact angle at the triple junction between a liquid, solid and surrounding air. Our discretization is based on the surface energy itself, rather than on the associated traction condition most commonly used for discretization with particle methods. Our energy based approach automatically captures surface gradients without the explicit need to resolve them as in traction condition based approaches. We include an implicit discretization of thermomechanical material coupling with a novel particle-based enforcement of Robin boundary conditions associated with convective heating. Lastly, we design a particle resampling approach needed to achieve perfect conservation of linear and angular momentum with Affine-Particle-In-Cell (APIC) [Jiang et al. 2015]. We show that our approach enables implicit time stepping for complex behaviors like the Marangoni effect and hydrophobicity/hydrophilicity. We demonstrate the robustness and utility of our method by simulating materials that exhibit highly diverse degrees of surface tension and thermomechanical effects, such as water, wine and wax.

Foundation paper piecing is a popular technique for constructing fabric patchwork quilts using printed paper patterns. But, the construction process imposes constraints on the geometry of the pattern and the order in which the fabric pieces are attached to the quilt. Manually designing foundation paper pieceable patterns that meet all of these constraints is challenging. In this work we mathematically formalize the foundation paper piecing process and use this formalization to develop an algorithm that can automatically check if an input pattern geometry is foundation paper pieceable. Our key insight is that we can represent the geometric pattern design using a certain type of dual hypergraph where nodes represent faces and hyperedges represent seams connecting two or more nodes. We show that determining whether the pattern is paper pieceable is equivalent to checking whether this hypergraph is acyclic, and if it is acyclic, we can apply a leaf-plucking algorithm to the hypergraph to generate viable sewing orders for the pattern geometry. We implement this algorithm in a design tool that allows quilt designers to focus on producing the geometric design of their pattern and let the tool handle the tedious task of determining whether the pattern is foundation paper pieceable.

Virtual Reality (VR) controllers are widely used for easy object selection and manipulation as a primary 3D input method in the virtual environment. Mobile devices with touchscreens like smartphones or tablets provide precise 2D tangible inputs. This research combines a VR controller and a touch-based smartphone to create a novel hybrid 2D-3D interface for enhanced VR interaction. We present the interface design and its implementation and also demonstrate four featured scenarios with the hybrid interface.

In full-color inkjet 3D printing, a key problem is determining the material configuration for the millions of voxels that a printed object is made of. The goal is a configuration that minimises the difference between desired target appearance and the result of the printing process. So far, the techniques used to find such a configuration have relied on domain-specific methods or heuristic optimization, which allowed only a limited level of control over the resulting appearance. We propose to use differentiable volume rendering in a continuous material-mixture space, which leads to a framework that can be used as a general tool for optimising inkjet 3D printouts. We demonstrate the technical feasibility of this approach, and use it to attain fine control over the fabricated appearance, and high levels of faithfulness to the specified target.

Physically accurate rendering often calls for taking the wave nature of light into consideration. In computer graphics, this is done almost exclusively locally, i.e. on a micrometre scale where the diffractive phenomena arise. However, the statistical properties of light, that dictate its coherence characteristics and its capacity to give rise to wave interference effects, evolve globally: these properties change on, e.g., interaction with a surface, diffusion by participating media and simply by propagation. In this paper, we derive the first global light transport framework that is able to account for these properties of light and, therefore, is fully consistent with Maxwell's electromagnetic theory. We show that our framework is a generalization of the classical, radiometry-based light transport---prominent in computer graphics---and retains some of its attractive properties. Finally, as a proof of concept, we apply the presented framework to a few practical problems in rendering and validate against well-studied methods in optics.

We present a fitted model of sky dome radiance and attenuation for realistic terrestrial atmospheres. Using scatterer distribution data from atmospheric measurement data, our model considerably improves on the visual realism of existing analytical clear sky models, as well as of interactive methods that are based on approximating atmospheric light transport. We also provide features not found in fitted models so far: radiance patterns for post-sunset conditions, in-scattered radiance and attenuation values for finite viewing distances, an observer altitude resolved model that includes downward-looking viewing directions, as well as polarisation information. We introduce a fully spherical model for in-scattered radiance that replaces the family of hemispherical functions originally introduced by Perez et al., and which was extended for several subsequent analytical models: our model relies on reference image compression via tensor decomposition instead.

Consistently exhibited personalities are crucial elements of realistic, engaging, and behavior-rich conversational virtual agents. Both nonverbal and verbal cues help convey these agents’ unseen psychological states, contributing to our effective communication with them. We introduce a comprehensive framework to design conversational agents that express personality through non-verbal behaviors like body movement and facial expressions, as well as verbal behaviors like dialogue selection and voice transformation. We use the OCEAN personality model, which defines personality as a combination of five orthogonal factors of openness, conscientiousness, extraversion, agreeableness, and neuroticism. The framework combines existing personality expression methods with novel ones such as new algorithms to convey Laban Shape and Effort qualities. We perform Amazon Mechanical Turk studies to analyze how different communication modalities influence our perception of virtual agent personalities and compare their individual and combined effects on each personality dimension. The results indicate that our personality-based modifications are perceived as natural, and each additional modality improves perception accuracy, with the best performance achieved when all the modalities are present. We also report some correlations for the perception of conscientiousness with neuroticism and openness with extraversion.

Basket weaving is a traditional craft for creating curved surfaces as an interwoven array of thin, flexible, and initially straight ribbons. The three-dimensional shape of a woven structure emerges through a complex interplay of the elastic bending behavior of the ribbons and the contact forces at their crossings. Curvature can be injected by carefully placing topological singularities in the otherwise regular weaving pattern. However, shape control through topology is highly non-trivial and inherently discrete, which severely limits the range of attainable woven geometries. Here, we demonstrate how to construct arbitrary smooth free-form surface geometries by weaving carefully optimized curved ribbons. We present an optimization-based approach to solving the inverse design problem for such woven structures. Our algorithm computes the ribbons' planar geometry such that their interwoven assembly closely approximates a given target design surface in equilibrium. We systematically validate our approach through a series of physical prototypes to show a broad range of new woven geometries that is not achievable by existing methods. We anticipate our computational approach to significantly enhance the capabilities for the design of new woven structures. Facilitated by modern digital fabrication technology, we see potential applications in material science, bio- and mechanical engineering, art, design, and architecture.