SIGGRAPH 2020 论文列表

In this paper, we present the concept of Remote Empathetic Viewpoints, an approach which allows the viewer to simultaneously access multiple remote viewpoints, granting us the ability to explore and extend the concepts of Cubism in 3D and in animation. In Remote Empathetic Viewpoints, we utilize a single Primary View Camera, a small set of Control lights, and a significantly large number of secondary cameras, whose positions and directions are referenced by Control Lights. By using such a multi camera system, we overcome the “shower door” affect that comes from using multiple cameras, which are used to obtain cubist rendering. By animating camera movement, we obtain temporal cubist art.

We present a valid polarization-based reflection contaminated image synthesis method, which can provide adequate, diverse and authentic training dataset. Meanwhile, we enhance the neural network by introducing the reflection information as guidance and utilizing adaptive convolution kernel size to fuse multi-scale information. We demonstrate that the proposed approach achieves convincing improvements over state of the arts.

In this research, we propose a digital fabrication method that can print reusable objects rapidly using the clay material. Therefore, we developed a magnetic force control device and a magnetic thermoplastic material that deforms dynamically. When the user creates a shape model on the screen, the clay material instantly changes to the shape of the designed model. The user can confirm the shape in a short time, and the material can be reused. In this paper, we describe the design and implementation of magnetic materials and control devices, evaluation, and future work.

Procedural terrain generation refers to the generation of terrain features, such as landscaping, rivers or road networks, through the use of algorithms, with minimal input required from the user. In the process of game development, generating terrain is often an important part of the game development process. Traditional generation methods are often too time consuming especially with larger terrain maps. On the other hand, procedural methods that generate terrain automatically often do not have much user control over the output. We explore the usage of conditional generative adversarial networks in the creation of road maps, as well as the application of such road maps in the creation of game levels in game development engines such as Unreal Engine 4.

We present a color mapping method that corresponds to intensity and considers the interaction of light among objects in a 3-dimensional scene. Previous methods that map color according to geometric specifications or a rendered result do not reproduce the interreflection of surrounding objects. Using a path tracer, the proposed method replaces the radiance with an arbitrary one-dimensional texture in the sampled light transport path. The texture appearing on the object is calculated by a weighted average of the modified estimations. The proposed method can generate color variation in physically based intensity. It hence can reproduce physical events such as caustics or color bleeding while retaining the flexibility of rendering within a path tracing framework.

We estimate depth maps from real-life monocular 360º VR videos using a Deep Neural Net architecture trained on 3D point-based renderings of people (called depth “mannequins”) captured with a Multi36 camera setup and processed with a combined pipeline of AI Instance Segmentation, Structure from Motion and Multi View Stereo methods.

To understand the sound propagation in a real space, we propose the mixed reality (MR) visualization system of instantaneous sound intensity, which can be measured at each measurement point with moving a handy microphone array in synchronization with sound reproduction. By visualizing the measured instantaneous sound intensities with an MR animation, we can observe the temporal flow of sound energy in the real space to understand the propagation of sound wave and acoustic properties, such as sound reflection.

To smoothly operate an HMD controller in the 3D space, a variety of operation methods that are linked to the real hand movements have been proposed. Many controller recognition methods are based on outside-in tracking. Therefore, those controllers have the disadvantage that it cannot be operated if it moves into the occluded area from the sensor. In this paper, we propose an inside-out controller for HMD using a smartphone that can estimate its own position and the angle via image recognition. This is a system that transmits the controller’s 6DoF(six degrees of freedom) information to the HMD and performs raycasting based on the 6DoF information. Wherever the controller is located, the user can operate it. This method covers the disadvantages of existing HMD controllers.

As a new method that can contribute to 3D shape analysis, we propose Incremental Contour Flow (ICF), which divides a 3D object into segments separated by boundary surfaces at narrow parts. ICF utilizes a property of distance transform that generates local maxima in a center of swollen part and bottleneck-like structures in a narrow part. Core of segment is defined as a local maximum layer that has a higher value than surrounding layers in a layer structure generated by converting the distance values to integers. Voxels in a surrounding layer of a core are added to the core incrementally, thereby 3D voxels are grouped to as many segments as the cores. As shown in an example to be discussed, ICF can generates combined soap bubble-like boundary surfaces from a 3D object made from merged three spheres. Since human organs tend to be distinguished by its shape in medical 3D images, ICF can be useful in the medical imaging applications.

For a foreground object in motion, details of its background which would otherwise be hidden are uncovered through its inner blur. This paper presents a novel hybrid motion blur rendering technique combining post-process image filtering and hardware-accelerated ray tracing. In each frame, we advance rays recursively into the scene to retrieve background information for inner blur regions and apply a post-process filtering pass on the ray-traced background and rasterized colour before compositing them together. Our approach achieves more accurate partial occlusion semi-transparencies for moving objects while maintaining interactive frame rates.

Depth of Field (DoF) in games is usually achieved as a post-process effect by blurring pixels in the sharp rasterized image based on the defined focus plane. This paper describes a novel real-time DoF technique that uses ray tracing with image filtering to achieve more accurate partial occlusion semi-transparencies on edges of blurry foreground geometry. This hybrid rendering technique leverages ray tracing hardware acceleration as well as spatio-temporal reconstruction techniques to achieve interactive frame rates.

We propose a GAN(Generative Adversarial Networks)-based drawing board which takes the semantic (by segmentation) and color tone (by strokes) inputs from users and automatically generates paintings. Our approach is built on a novel and lightweight feature embedding which incorporates the colorization effects into the painting generation process. Unlike the existing GAN-based image generation models which take semantics input, our drawing board has the ability to edit the local colors after generation. Our method samples the color information from users’ strokes as extra input, then feeds it into a GAN model for conditional generation. We enable the creation of pictures or paintings with semantics and color control in real-time.

In this paper we explore how different interaction types (POV, cloth, visual effects, and background music) can influence the user experiences in terms of interest, immersion, fun, and ease of use. To this end, we design an interactive virtual reality fashion show system with a pipeline approach and develop it using Unity3D game engine and HTC Vive HMD. A pilot study with 12 participants shows that the four interaction types can positively affect the user experiences.

1 EVALUATION OF A VIRTUAL REALITY-BASED BUFFET TO ADDRESS CHALLENGES IN HEALTH RESEARCH AND PRACTICE The Immersive Virtual Environment Testing Area (IVETA) is a behavioral science laboratory offering tools and expert support for virtual reality (VR)-based health research at the National Institutes of Health. We design VR tools to tackle methodological and communication challenges in the domain of health research and practice, with a specialty in genomics. One of our most useful VR tools is a custom virtual reality application intended to simulate a buffet food choice experience in a typical cafeteria setting. Food choice practices are a crucial research topic, but it remains difficult to assess as few methods exist to measure choices and their nutritional content accurately in real-time. The simulation is typically used to conduct experiments on child feeding, involving parents of children between the ages of 3 and 7. Participants walk freely around the buffet, navigating food choices in an environment designed specifically to fit in the bounds of our lab space. The VR buffet was designed initially for legacy virtual reality hardware but has since been through several iterations and is primarily to be used with the readily available HTC Vive system. Given that most of our study participants lacked any experience with virtual reality, and our goal was to simulate a familiar, real world activity, important design decisions for ease of use needed to be made from the start. Therefore, the input mapping for the Vive controllers was made to be dead simple: with a single click, participants could either add or subtract one portion of food to or from their plate.

Figure 1: Left: Orientation-based redirection in Virtual Reality (VR) using dynamic-foveated rendering which leverages the effect of inattentional blindness induced by a cognitive task. The blending (e.g. parallax of green alien) in the transition zone between the rotated virtual environment (VE) (non-foveal) and the non-rotated VE (foveal) is imperceptible by the users due to inattentional blindness. Angular gain of the non-foveal zone for the frame shown is at its maximum i.e. 13.5°. Right: The path walked in physical tracked space (PTS) up to that point is shown in orange color. The corresponding path in the VE is shown in blue color. The cyan-colored box indicates the 4 × 4m2 of available PTS and the camera icon inside the box indicates the location of the user w.r.t. the PTS. Even with room-scale PTS such as the one used, the users were able to walk distances in the virtual environment (VE) which are up to almost 18 times higher than the longest distance in the PTS; maximum recorded in experiments was 103.9m.