SIGGRAPH Asia 2022 Papers

We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape, texture or both. To constrain the optimization to produce plausible meshes and textures we introduce a number of techniques using image augmentations and the use of a pretrained prior that generates CLIP image embeddings given a text embedding.

Concept sketches are ubiquitous in industrial design, as they allow designers to quickly depict imaginary 3D objects. To construct their sketches with accurate perspective, designers rely on longstanding drawing techniques, including the use of auxiliary construction lines to identify midpoints of perspective planes, to align points vertically and horizontally, and to project planar curves from one perspective plane to another. We present a method to synthesize such construction lines from CAD sequences. Importantly, our method balances the presence of construction lines with overall clutter, such that the resulting sketch is both well-constructed and readable, as professional designers are trained to do. In addition to generating sketches that are visually similar to real ones, we apply our method to synthesize a large quantity of paired sketches and normal maps, and show that the resulting dataset can be used to train a neural network to infer normals from concept sketches.1

This work introduces a novel method for automating the object animation from a 3D mesh used in various applications such as AR/VR, gaming, etc. The 3D mesh could be a generic inflated object generated from an image or sketch. The method describes a medial axis based control point estimation to generate an animation path and perform deformation-based animations for a mesh. Our method eliminates user involvement or need for expertise in terms of rigging the mesh or region selection etc. Performance of our method shows meaningful, real-time animation of the 3D mesh. Experiments indicate that our method is more consistent and less error prone compared to existing works.

When conducting autonomous scanning for the online reconstruction of unknown indoor environments, robots have to be competent at exploring scene structure and reconstructing objects with high quality. Our key observation is that different tasks demand specialized scanning properties of robots: rapid moving speed and far vision for global exploration and slow moving speed and narrow vision for local object reconstruction, which are referred as two different scanning modes: explorer and reconstructor, respectively. When requiring multiple robots to collaborate for efficient exploration and fine-grained reconstruction, the questions on when to generate and how to assign those tasks should be carefully answered. Therefore, we propose a novel asynchronous collaborative autoscanning method with mode switching, which generates two kinds of scanning tasks with associated scanning modes, i.e., exploration task with explorer mode and reconstruction task with reconstructor mode, and assign them to the robots to execute in an asynchronous collaborative manner to highly boost the scanning efficiency and reconstruction quality. The task assignment is optimized by solving a modified Multi-Depot Multiple Traveling Salesman Problem (MDMTSP). Moreover, to further enhance the collaboration and increase the efficiency, we propose a task-flow model that actives the task generation and assignment process immediately when any of the robots finish all its tasks with no need to wait for all other robots to complete the tasks assigned in the previous iteration. Extensive experiments have been conducted to show the importance of each key component of our method and the superiority over previous methods in scanning efficiency and reconstruction quality.

Assembly planning is the core of automating product assembly, maintenance, and recycling for modern industrial manufacturing. Despite its importance and long history of research, planning for mechanical assemblies when given the final assembled state remains a challenging problem. This is due to the complexity of dealing with arbitrary 3D shapes and the highly constrained motion required for real-world assemblies. In this work, we propose a novel method to efficiently plan physically plausible assembly motion and sequences for real-world assemblies. Our method leverages the assembly-by-disassembly principle and physics-based simulation to efficiently explore a reduced search space. To evaluate the generality of our method, we define a large-scale dataset consisting of thousands of physically valid industrial assemblies with a variety of assembly motions required. Our experiments on this new benchmark demonstrate we achieve a state-of-the-art success rate and the highest computational efficiency compared to other baseline algorithms. Our method also generalizes to rotational assemblies (e.g., screws and puzzles) and solves 80-part assemblies within several minutes.

This paper presents a procedural modeling method for coral groups considering the territorial conflict between species. We developed a graphical interface to control the territorial battle by arranging locator points that represent available space to grow the coral branches. The coral skeletals are generated according to species-specific structural rules while competing for the locators between individuals.

In 3D character animation, the frame rate is often reduced to mimic hand-drawn animations, like anime (Japanese animation). However, anime with a low frame rate differs from real motions because it omits excessive movements and emphasizes speed by expressing motions with a small number of impressive poses. It is difficult to reproduce such motions only by downsampling mocap data. Thus, in this poster, we propose a method for converting mocap data into anime-like 2D motions by respecting production site techniques. The proposed method evaluates the characteristics of motion data using the time distributions of speeds and pose areas to select an appropriate sequence of viewpoints and extract effective poses for each viewpoint.

We present Animatomy, a novel anatomic+animator centric representation of the human face. Present FACS-based systems are plagued with problems of face muscle separation, coverage, opposition, and redundancy. We, therefore, propose a collection of muscle fiber curves as an anatomic basis, whose contraction and relaxation provide us with a fine-grained parameterization of human facial expression. We build an end-to-end modular deformation architecture using this representation that enables: automatic optimization of the parameters of a specific face from high-quality dynamic facial scans; face animation driven by performance capture, keyframes, or dynamic simulation; interactive and direct manipulation of facial expression; and animation transfer from an actor to a character. We validate our facial system by showing compelling animated results, applications, and a quantitative comparison of our facial reconstruction to ground truth performance capture. Our system is being intensively used by a large creative team on Avatar: The Way of Water. We report feedback from these users as qualitative evaluation of our system.

3D Morphable models of the human body capture variations among subjects and are useful in reconstruction and editing applications. Current dental models use an explicit mesh scene representation and model only the teeth, ignoring the gum. In this work, we present the first parametric 3D morphable dental model for both teeth and gum. Our model uses an implicit scene representation and is learned from rigidly aligned scans. It is based on a component-wise representation for each tooth and the gum, together with a learnable latent code for each of such components. It also learns a template shape thus enabling several applications such as segmentation, interpolation and tooth replacement. Our reconstruction quality is on par with the most advanced global implicit representations while enabling novel applications. The code will be available at https://github.com/cong-yi/DMM

Stylized 3D avatars have become increasingly prominent in our modern life. Creating these avatars manually usually involves laborious selection and adjustment of continuous and discrete parameters and is time-consuming for average users. Self-supervised approaches to automatically create 3D avatars from user selfies promise high quality with little annotation cost but fall short in application to stylized avatars due to a large style domain gap. We propose a novel self-supervised learning framework to create high-quality stylized 3D avatars with a mix of continuous and discrete parameters. Our cascaded domain bridging framework first leverages a modified portrait stylization approach to translate input selfies into stylized avatar renderings as the targets for desired 3D avatars. Next, we find the best parameters of the avatars to match the stylized avatar renderings through a differentiable imitator we train to mimic the avatar graphics engine. To ensure we can effectively optimize the discrete parameters, we adopt a cascaded relaxation-and-search pipeline. We use a human preference study to evaluate how well our method preserves user identity compared to previous work as well as manual creation. Our results achieve much higher preference scores than previous work and close to those of manual creation. We also provide an ablation study to justify the design choices in our pipeline.

We present a spectral measurement approach for the bulk optical properties of translucent materials using only low-cost components. We focus on the translucent inks used in full-color 3D printing, and develop a technique with a high spectral resolution, which is important for accurate color reproduction. We enable this by developing a new acquisition technique for the three unknown material parameters, namely, the absorption and scattering coefficients, and its phase function anisotropy factor, that only requires three point measurements with a spectrometer. In essence, our technique is based on us finding a three-dimensional appearance map, computed using Monte Carlo rendering, that allows the conversion between the three observables and the material parameters. Our measurement setup works without laboratory equipment or expensive optical components. We validate our results on a 3D printed color checker with various ink combinations. Our work paves a path for more accurate appearance modeling and fabrication even for low-budget environments or affordable embedding into other devices.

Recently, many image stylization techniques have been proposed to convert photographs into illustrations, paintings, cartoons, etc. Artists sometimes use these techniques to efficiently generate background images for their own illustrations of foreground objects, such as cartoon characters. However, this approach often produces unnatural results because the level of abstraction between the foreground and the background images are different, resulting in inconsistent atmosphere between them. In this paper, we address this problem and propose a method for adjusting the abstraction level of the stylized background images. The effectiveness of our method is demonstrated by several examples with different levels of abstraction.

Real-time visualization of large-scale surface models is still a challenging problem. When using multiple levels of details (LOD), the main issues are popping between levels and/or cracks between level parts. We present a scheme (both mesh preprocessing and real-time rendering), which avoids both of these issues. Vertex interpolation between child and parent is used to achieve crack and popping free rendering. We implemented and tested our method on a modest laptop PC, and could render scanned models of multiples tens of million triangles at optimal visual quality and interactive frame rate.

Large, high-resolution displays are installed throughout the city as public displays. By superimposing invisible information on the images of these displays, large numbers of devices with cameras and sensors can communicate with the displays without prior pairing. Several applications have been proposed, such as operating robots or communicating information to users by displaying 2D codes on images. However, the display of 2D codes has the problem of compromising the appearance of displayed content. Abe et al. proposed a method of communicating with devices by superimposing invisible information using color vibration on images displayed on off-the-shelf liquid-crystal displays (LCD). Using this method, we can embed the information for devices in images without interfering with the displayed content. Abe et al. uses a simple serial loop operation to search for color pairs comprising a color vibration, which requires a very long processing time due to the huge search space. In this paper, we propose an accelerated and optimized search method for color pairs that constitute the imperceptible color vibration for embedding information on LCD images. To achieve fast color pair search, we parallelized the search process, which is previously done individually, by using arrays representing the amount of movement and an operation to extract elements from the array that satisfy the conditions. In addition, we investigate the amount of information that can be superimposed on nine color images using the imperceptible color vibration and clarify the applicability of embedding information into images using the color vibration.

We address the problem of controlling and simulating interactions between multiple physics-based characters, using short unlabeled motion clips. We propose Adversarial Interaction Priors (AIP), a multi-agents generative adversarial imitation learning (MAGAIL) approach, which extends recent deep reinforcement learning (RL) works aiming at imitating single character example motions. The main contribution of this work is to extend the idea of motion imitation of a single character to interaction imitation between multiple characters. Our method uses a control policy for each character to imitate interactive behaviors provided by short example motion clips, and associates a discriminator for each character, which is trained on actor-specific interactive motion clips. The discriminator returns interaction rewards that measure the similarity between generated behaviors and demonstrated ones in the reference motion clips. The policies and discriminators are trained in a multi-agent adversarial reinforcement learning procedure, to improve the quality of the behaviors generated by each agent. The initial results show the effectiveness of our method on the interactive task of shadowboxing between two fighters.

People with visual impairment (PVI) are eager to push their limits in extreme sports such as alpine skiing. However, training skiing is very difficult as it always requires assistance from an experienced guide. This paper explores sonification-based methods that enable PVI to train skiing using a simulator, which will allow them to train without a guide. Two types of sonification feedback for PVI are proposed and studied in our experiment. The results suggest that users without any visual information can also pass through over 80% of the poles compared to with visual information on average.

Preoperative models manufactured by 3D printing technology are widely demanded in surgical procedures. Current printing methods such as Polyjet, color jet printing (CJP) and fused deposition molding (FDM) are either too expensive or of poor performance. In this paper, a novel solution is proposed. Preoperative models are first divided into internal part, such as blood vessel, and the main body. The former is printed in color and the latter is transparent. In particular, the internal part is printed using FDM. A housing template for the main body is generated by digital light processing (DLP) based on the preoperative model. After integrating the template and inner part, transparent elastomer precursor is injected and cured inside the template, leading to the formation of the transparent main body. For demonstration, a liver model is manufactured using the proposed scheme. Compared with the existing methods, our strategy could manufacture preoperative models at both high quality and affordable cost.

Soil structure describes the association and arrangement of the soil granules. Structured clays, such as the cracked dry ground, exhibit characteristics of solid-like materials, while destructured clays exhibit characteristics of granular materials. The structure strength weakens with the deviatoric plastic strain accumulation, which leads to clay degradation. However, the existing graphics methods for granular materials cannot simulate this unique mechanical behavior. We present a non-associated Modified Structured Cam Clay (MSCC) model for simulating structured and destructured clays. We track the deviatoric part of the plastic Hencky strain and design two degradation functions (corresponding to yield stress and mean effective stress) to control the size of the yield surface. Through a non-associated flow rule, our method generates visually plausible results while allowing volume preservation.