SIGGRAPH 2022 Papers

Active soft bodies can affect their shape through an internal actuation mechanism that induces a deformation. Similar to recent work, this paper utilizes a differentiable, quasi-static, and physics-based simulation layer to optimize for actuation signals parameterized by neural networks. Our key contribution is a general and implicit formulation to control active soft bodies by defining a function that enables a continuous mapping from a spatial point in the material space to the actuation value. This property allows us to capture the signal's dominant frequencies, making the method discretization agnostic and widely applicable. We extend our implicit model to mandible kinematics for the particular case of facial animation and show that we can reliably reproduce facial expressions captured with high-quality capture systems. We apply the method to volumetric soft bodies, human poses, and facial expressions, demonstrating artist-friendly properties, such as simple control over the latent space and resolution invariance at test time.

We present Immersive-Labeler, an environment for the annotation of large-scale 3D point cloud scenes of urban environments. Our concept is based on the full immersion of the user in a VR-based environment that represents the 3D point cloud scene while offering adapted visual aids and intuitive interaction and navigation modalities. Through a user-centric design, we aim to improve the annotation experience and thus reduce its costs. For the preliminary evaluation of our environment, we conduct a user study (N=20) to quantify the effect of higher levels of immersion in combination with the visual aids we implemented on the annotation process. Our findings reveal that higher levels of immersion combined with object-based visual aids lead to a faster and more engaging annotation process.

We aim to ask and answer an essential question "how quickly do we react after observing a displayed visual target?" To this end, we present psychophysical studies that characterize the remarkable disconnect between human saccadic behaviors and spatial visual acuity. Building on the results of our studies, we develop a perceptual model to predict temporal gaze behavior, particularly saccadic latency, as a function of the statistics of a displayed image. Specifically, we implement a neurologically-inspired probabilistic model that mimics the accumulation of confidence that leads to a perceptual decision. We validate our model with a series of objective measurements and user studies using an eye-tracked VR display. The results demonstrate that our model prediction is in statistical alignment with real-world human behavior. Further, we establish that many sub-threshold image modifications commonly introduced in graphics pipelines may significantly alter human reaction timing, even if the differences are visually undetectable. Finally, we show that our model can serve as a metric to predict and alter reaction latency of users in interactive computer graphics applications, thus may improve gaze-contingent rendering, design of virtual experiences, and player performance in e-sports. We illustrate this with two examples: estimating competition fairness in a video game with two different team colors, and tuning display viewing distance to minimize player reaction time.

Low-overlap regions between paired point clouds make the captured features very low-confidence, leading cutting edge models to point cloud registration with poor quality. Beyond the traditional wisdom, we raise an intriguing question: Is it possible to exploit an intermediate yet misaligned image between two low-overlap point clouds to enhance the performance of cutting-edge registration models? To answer it, we propose a misaligned image supported registration network for low-overlap point cloud pairs, dubbed ImLoveNet. ImLoveNet first learns triple deep features across different modalities and then exports these features to a two-stage classifier, for progressively obtaining the high-confidence overlap region between the two point clouds. Therefore, soft correspondences are well established on the predicted overlap region, resulting in accurate rigid transformations for registration. ImLoveNet is simple to implement yet effective, since 1) the misaligned image provides clearer overlap information for the two low-overlap point clouds to better locate overlap parts; 2) it contains certain geometry knowledge to extract better deep features; and 3) it does not require the extrinsic parameters of the imaging device with respect to the reference frame of the 3D point cloud. Extensive qualitative and quantitative evaluations on different kinds of benchmarks demonstrate the effectiveness and superiority of our ImLoveNet over state-of-the-art approaches.

We present Holographic Glasses, a holographic near-eye display system with an eyeglasses-like form factor for virtual reality. Holographic Glasses are composed of a pupil-replicating waveguide, a spatial light modulator, and a geometric phase lens to create holographic images in a lightweight and thin form factor. The proposed design can deliver full-color 3D holographic images using an optical stack of 2.5 mm thickness. A novel pupil-high-order gradient descent algorithm is presented for the correct phase calculation with the user’s varying pupil size. We implement benchtop and wearable prototypes for testing. Our binocular wearable prototype supports 3D focus cues and provides a diagonal field of view of 22.8° with a 2.3 mm static eye box and additional capabilities of dynamic eye box with beam steering, while weighing only 60 g excluding the driving board.

Holography is a promising avenue for high-quality displays without requiring bulky, complex optical systems. While recent work has demonstrated accurate hologram generation of 2D scenes, high-quality holographic projections of 3D scenes has been out of reach until now. Existing multiplane 3D holography approaches fail to model wavefronts in the presence of partial occlusion while holographic stereogram methods have to make a fundamental tradeoff between spatial and angular resolution. In addition, existing 3D holographic display methods rely on heuristic encoding of complex amplitude into phase-only pixels which results in holograms with severe artifacts. Fundamental limitations of the input representation, wavefront modeling, and optimization methods prohibit artifact-free 3D holographic projections in today’s displays. To lift these limitations, we introduce hogel-free holography which optimizes for true 3D holograms, supporting both depth- and view-dependent effects for the first time. Our approach overcomes the fundamental spatio-angular resolution tradeoff typical to stereogram approaches. Moreover, it avoids heuristic encoding schemes to achieve high image fidelity over a 3D volume. We validate that the proposed method achieves 10 dB PSNR improvement on simulated holographic reconstructions. We also validate our approach on an experimental prototype with accurate parallax and depth focus effects.

Barrier-grid animation is a primitive animation technique that utilizes an interlaced image with a transparent striped overlay. Ombro-Cinéma, a toy developed in 1921, utilizes this technique effectively with a scroll of graphics. By rotating an attached hand-crank, it presents a scrolling animated story, the narrative of which is often tightly associated with its mechanical structure. This paper describes High-Low Tech Ombro-Cinéma, a prototype device that extends Ombro-Cinéma using an embedded system. Conductive adhesive tape strips are affixed to the back side of a scroll and used to trigger various sonic events to enhance its narrativity. Such an investigation into the reproduction of an obsolete toy with digital technology may provide an interesting opportunity to review the narrative of an obsolete device and to reconsider the effects tangible interfaces have on narratives.

Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our knowledge) to the reconstruction of highresolution, high-dynamic range color images from raw photographic bursts captured by a handheld camera with exposure bracketing. This method uses a physically-accurate model of image formation to combine an iterative optimization algorithm for solving the corresponding inverse problem with a learned image representation for robust alignment and a learned natural image prior. The proposed algorithm is fast, with low memory requirements compared to state-of-the-art learning-based approaches to image restoration, and features that are learned end to end from synthetic yet realistic data. Extensive experiments demonstrate its excellent performance with super-resolution factors of up to ×4 on real photographs taken in the wild with hand-held cameras, and high robustness to low-light conditions, noise, camera shake, and moderate object motion.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). SIGGRAPH ’22 Posters, August 07-11, 2022, Vancouver, BC, Canada © 2022 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9361-4/22/08. https://doi.org/10.1145/3532719.3543226 users’ browsing habits. For example, if a user wants to find a certain approach at an academic conference, they will most likely obtain multiple results with video thumbnail and title after long-time searching. These interfaces may not allow users to intuitively know the video contents. We propose a video browsing interface, HiVideo, for academic presentations and educational videos, as shown in Fig. 1. When users retrieve the specified contents, the poster layer can help them explore the general video content rather than watching the whole video. To extract slides from a video, the proposed system leverages a frame detection approach, and a deep learning based document layout analysis approach [Shen et al. 2021] to extract visual information from the slides. In the user study, we verify that this proposed HiVideo interface can accommodate a wide range of usage scenarios for different users. In contrast to a previous video browsing interface [Weng et al. 2022], we adopt responsive web design so that users can achieve the better user experience even with mobile browsers.

Figure 1: A) HYFAR for haptic clothing interfaces. B) Uniaxial tensile test of HYFAR, its constitutive materials, and finite element (FE) simulation. C) Experimental data and simulation of the inflation of a circular actuator. D) Mapping of material parameter matrices to the FE mesh for an actuator embroidered with a cross-pattern (c.p.), and simulation results. E) Simulation ofHYFARwith various embroidered patterns inflated at 40 kPa. F) Blocked force versus bending angle for stadium-shaped HYFARs integrated in a unidirectional joint. Upper-body functional garments integrated with HYFAR G) (Design B) for kinesthetic haptic feedback, and with H) circular actuator (r = 4.5 cm; c.p. and four LEDs) for showcasing shape shifting behavior.

Reconstruction of high-fidelity 3D objects or scenes is a fundamental research problem. Recent advances in RGB-D fusion have demonstrated the potential of producing 3D models from consumer-level RGB-D cameras. However, due to the discrete nature and limited resolution of their surface representations (e.g., point or voxel based), existing approaches suffer from the accumulation of errors in camera tracking and distortion in the reconstruction, which leads to an unsatisfactory 3D reconstruction. In this article, we present a method using on-the-fly implicits of Hermite Radial Basis Functions (HRBFs) as a continuous surface representation for camera tracking in an existing RGB-D fusion framework. Furthermore, curvature estimation and confidence evaluation are coherently derived from the inherent surface properties of the on-the-fly HRBF implicits, which are devoted to a data fusion with better quality. We argue that our continuous but on-the-fly surface representation can effectively mitigate the impact of noise with its robustness and constrain the reconstruction with inherent surface smoothness when being compared with discrete representations. Experimental results on various real-world and synthetic datasets demonstrate that our HRBF-fusion outperforms the state-of-the-art approaches in terms of tracking robustness and reconstruction accuracy.

We present a technique to reduce the dynamic range of an HDRI lighting environment map in an efficient, energy-preserving manner by spreading out the light of concentrated light sources. This allows us to display a reasonable approximation of the illumination of an HDRI map in a lighting reproduction system with limited dynamic range such as virtual production LED Stage. The technique identifies regions of the HDRI map above a given pixel threshold, dilates these regions until the average pixel value within each is below the threshold, and finally replaces each dilated region’s pixels with the region’s average pixel value. The new HDRI map contains the same energy as the original, spreads the light as little as possible, and avoids chromatic fringing.

We present a method for enhancing fluid simulations with realistic bubble and foam detail. We treat bubbles as discrete air particles, two-way coupled with a sparse volumetric Euler flow, as first suggested in [Stomakhin et al. 2020]. We elaborate further on their scheme and introduce a bubble inertia correction term for improved convergence. We also show how one can add bubbles to an already existing fluid simulation using our novel guiding technique, which performs local re-simulation of fluid to achieve more interesting bubble dynamics through coupling. As bubbles reach the surface, they are converted into foam and simulated separately. Our foam is discretized with smoothed particle hydrodynamics (SPH), and we replace forces normal to the fluid surface with a fluid surface manifold advection constraint to achieve more robust and stable results. The SPH forces are derived through proper constitutive modeling of an incompressible viscous liquid, and we explain why this choice is appropriate for "wet" types of foam. This allows us to produce believable dynamics from close-up scenarios to large oceans, with just a few parameters that work intuitively across a variety of scales. Additionally, we present relevant research on air entrainment metrics and bubble distributions that have been used in this work.

Partial differential equations (PDEs) with spatially varying coefficients arise throughout science and engineering, modeling rich heterogeneous material behavior. Yet conventional PDE solvers struggle with the immense complexity found in nature, since they must first discretize the problem---leading to spatial aliasing, and global meshing/sampling that is costly and error-prone. We describe a method that approximates neither the domain geometry, the problem data, nor the solution space, providing the exact solution (in expectation) even for problems with extremely detailed geometry and intricate coefficients. Our main contribution is to extend the walk on spheres (WoS) algorithm from constant- to variable-coefficient problems, by drawing on techniques from volumetric rendering. In particular, an approach inspired by null-scattering yields unbiased Monte Carlo estimators for a large class of 2nd order elliptic PDEs, which share many attractive features with Monte Carlo rendering: no meshing, trivial parallelism, and the ability to evaluate the solution at any point without solving a global system of equations.

Previous works have done several kinds of haptic techniques for simulating the touching experience of the virtual object. However, the feedback on the object’s weight has less been explored. This paper presents GravityPack, a wearable gravity display to simulate grabbing, holding, and releasing the virtual object in the virtual world using the liquid-based system consisting of pumps, pipes, valves, a water tank, and water packs. This system can provide a wide weight range from 110g to 1.8 kg in 40 seconds. Additionally, we design and investigate the visual feedback of weight transition for the delay time of liquid transfer to understand the feasibility of visualization by a user study. With the revealing of design consideration and implementation, the paper also shows the potential use of the liquid-based system and its possibility of the visualization technique to simulate the weight sensations.