IROS 2021 Papers

We present a Virtual Kinematic Chain (VKC) perspective, a simple yet effective method, to improve task planning efficacy for mobile manipulation. By consolidating the kinematics of the mobile base, the arm, and the object being manipulated collectively as a whole, this novel VKC perspective naturally defines abstract actions and eliminates unnecessary predicates in describing intermediate poses. As a result, these advantages simplify the design of the planning domain and significantly reduce the search space and branching factors in solving planning problems. In experiments, we implement a task planner using Planning Domain Definition Language (PDDL) with VKC. Compared with conventional domain definition, our VKC-based domain definition is more efficient in both planning time and memory. In addition, abstract actions perform better in producing feasible motion plans and trajectories. We further scale up the VKC-based task planner in complex mobile manipulation tasks. Taken together, these results demonstrate that task planning using VKC for mobile manipulation is not only natural and effective but also introduces new capabilities.

This paper presents a motion planning algorithm that enables robots to efficiently pick up objects by considering simultaneous multi-object grasping. At the center of the algorithm is a cost function that helps to determine one of the following three grasping policies considering distance and friction constraints – Grasping a single object; Grasping two objects simultaneously; Grasping two object simultaneously after pushing one of the objects close to the other. After recognizing the object distributions on a table by using a depth camera and Mask R-CNN, our algorithm will select grasp policies from the three candidates considering the cost function, and plan a policy sequence that can most quickly finish picking all the objects using dynamic programming. Both simulation and real-world experiments are carried out to examine the performance of the proposed planner. Results show that the proposed method significantly improves the efficiency of robotic picking compared to conventional single-object-based methods.

Due to the presence of ambiguities caused by sensor noise and structural similarity, simultaneous localization and mapping (SLAM) observation models are typically multimodal. The multimodal inference process can be directly dealt with by belief propagation (BP) using weighted Gaussian mixture messages, but for efficiency, a combinatorial explosion of the complexity must be suitably relaxed. In this study, we present an effective multimodal BP SLAM for robust inference with ambiguities. Using Gaussian bandwidth mean shift and cluster-based reparameterization, we reduce the number of Gaussian components in each message due to the BP nature. The proposed algorithm reduces the number of components of the product by summarizing indistinguishable modes in weighted Gaussian mixtures and keeping only the significant modes, making BP computationally efficient.

In recent times, quadrotors have become immensely applicable in scenarios such as relief operations, infrastructure maintenance, search-and-rescue missions etc. A key control design challenge arises in these applications when the quadrotor has to manoeuvre through constrained spaces such as narrow windows, pipelines in the presence of external disturbances and parametric uncertainties: such conditions necessitate the controller to guarantee predefined tracking accuracy so as to not violate the constraints and simultaneously tackle uncertainties. However, state-of-the-art controllers dealing with constrained system motion are not applicable either for an underactuated system like quadrotor or for an uncertain system dynamics. This work proposes a robust controller that enables the quadrotor to follow a trajectory with predefined tracking accuracy in constrained space as well as to tackle uncertainties stemming from imprecise system modelling and external disturbances. The closed-loop system stability is analysed via the Barrier Lyapunov approach and the effectiveness of the proposed controller is validated via simulation with state of the art.

The ability to localise is key for robot navigation. We describe an efficient method for vision-based localisation, which combines sequential Monte Carlo tracking with matching ground-level images to 2-D cartographic maps such as OpenStreetMaps. The matching is based on a learned embedded space representation linking images and map tiles, encoding the common semantic information present in both and providing potential for invariance to changing conditions. Moreover, the compactness of 2-D maps supports scalability. This contrasts with the majority of previous approaches based on matching with single-shot geo-referenced images or 3-D reconstructions. We present experiments using the StreetLearn and Oxford RobotCar datasets and demonstrate that the method is highly effective, giving high accuracy and fast convergence.

Identifying in-water obstacles is fundamental for safe navigation of Autonomous Surface Vehicles (ASVs). This paper presents a model-free method for segmenting individual in-water objects (e.g., swimmers, buoys, boats) and shorelines from LiDAR sensor data. To reduce the computational requirement, our method first converts the 3D point cloud into a 2D spherical projection image. Then, an algorithm based on the integration of a breadth-first search and a variant of a hierarchical agglomerative clustering segments the points according to different objects. Our method addresses the sparsity and instability of the point cloud in the aquatic domain – a characteristic that makes the methods developed for self-driving cars not directly applicable for in-water obstacle segmentation, as demonstrated in our experiments. Our method is compared with other state-of-the-art approaches and is validated both in simulation and in real-world ASV deployments, with different objects and encountering scenarios. The proposed method is effective in segmenting in-water obstacles not known a priori, in real-time, outperforming other state-of-the art methods.

Exploration tasks are essential to many emerging robotics applications, ranging from search and rescue to space exploration. The planning problem for exploration requires determining the best locations for future measurements that will enhance the fidelity of the map, for example, by reducing its total entropy. A widely-studied technique involves computing the Mutual Information (MI) between the current map and future measurements, and utilizing this MI metric to decide the locations for future measurements. However, computing MI for reasonably-sized maps is slow and power hungry, which has been a bottleneck towards fast and efficient robotic exploration. In this paper, we introduce a new hardware accelerator architecture for MI computation that features a low-latency, energy-efficient MI compute core and an optimized memory subsystem that provides sufficient bandwidth to keep the cores fully utilized. The core employs interleaving to counter the recursive algorithm, and workload balancing and numerical approximations to reduce latency and energy consumption. We demonstrate this optimized architecture with a Field-Programmable Gate Array (FPGA) implementation, which can compute MI for all cells in an entire 201-by-201 occupancy grid (e.g., representing a 20.1m-by-20.1m map at 0.1m resolution) in 1.55 ms while consuming 1.7 mJ of energy, thus finally rendering MI computation for the whole map real time and at a fraction of the energy cost of traditional compute platforms. For comparison, this particular FPGA implementation running on the Xilinx Zynq-7000 platform is two orders of magnitude faster and consumes three orders of magnitude less energy per MI map compute, when compared to a baseline GPU implementation running on an NVIDIA GeForce GTX 980 platform. The improvements are more pronounced when compared to CPU implementations of equivalent algorithms.

A non-lexical backchannel is known to be dependent on the conversational context, and its form can be distinguished by the social relation between the speaker and the listener in the Korean language. Thus, to investigate the effect of a non-lexical backchannel, we conducted a 2 (context: information-centric versus emotion-centric) × 3 (forms of backchannel: "ne" versus "eo" versus "eum") mixed-participant experiment (N = 96). After watching video stimuli, the participants evaluated a robot using one of the three forms of non-lexical backchannels under both contexts. No significant main effect of the forms of the non-lexical backchannel was found. By contrast, we found that task attraction and appropriateness of the robot were rated more positively under the information-centric context than under the emotion-centric context. The functions of the non-lexical backchannel of attentive listening and understanding were manifested more under the information-centric context than under the emotion-centric context. Furthermore, these functions mediated the effect of conversational context on the user perception of the robot in terms of task attraction and appropriateness.

With the recent diversification of operating devices, the demand for input operations that require confirmation of the effect of differences in display response on operability has increased. Regarding display response, previous studies have investigated the threshold time and sense of agency for a delayed response during device operation. However, these studies only focused on subjective evaluations. Therefore, this study aims to clarify the human motor characteristics and activated brain regions based on the differences in display response time during device operation. The target motion is the rotational operation of the cylindrical rotary controller using the index finger and thumb. The experimental conditions involve four types of display response times (the duration from the operation to the indicated response). We measured the brain activity using near-infrared spectroscopy, the muscle activity from a surface myoelectric potential measurement device, and the force data of the index finger and thumb tip obtained from two independent six-axis force/torque sensors. Although the experimental results showed no significant difference in the muscle activity and gripping force, a significant difference was observed in the brain activity and the questionnaire survey by the difference in display response time. This investigation reveals that the difference in display response time affects brain activity and subjective information, clarifying the relationship between brain activity and subjective information.

Robotic In-space assembly (ISA) is the next step to building larger and more permanent structures in orbit. Robotic ISA offers a unique opportunity for engineers to design the robotic system and the structure at the same time. ISA structures can be optimized to minimize weight or the number of pieces but these decisions have large impacts on the complexity of the robotic system. This impact goes beyond just defining the length and number of joints for the robotic system; the assembly process itself will drive the robot design. Robot trajectories will result in the forces and torques being applied to individual truss pieces and to the whole assembly itself, which are driven by the assembly plan. This study focuses on some of the design concerns of a robotic ISA system; specifically focusing on a walking robotic as it assembles a linear truss.

The broad scope of obstacle avoidance has led to many kinds of computer vision-based approaches. Despite its popularity, it is not a solved problem. Traditional computer vision techniques using cameras and depth sensors often focus on static scenes, or rely on priors about the obstacles. Recent developments in bio-inspired sensors present event cameras as a compelling choice for dynamic scenes. Although these sensors have many advantages over their frame-based counterparts, such as high dynamic range and temporal resolution, event-based perception has largely remained in 2D. This often leads to solutions reliant on heuristics and specific to a particular task.We show that the fusion of events and depth overcomes the failure cases of each individual modality when performing obstacle avoidance. Our proposed approach unifies event camera and lidar streams to estimate metric Time-To-Impact (TTI) without prior knowledge of the scene geometry or obstacles. In addition, we release an extensive event-based dataset with six visual streams spanning over 700 scanned scenes.

This study presents dynamics computation and control of a hybrid multi-link system that integrates rigid- and soft-bodies. It is a challenging problem to install a softness in a robot system, which is an important factor in human body. Softness achieved by human muscles and ligaments contributes to dynamic motion. Flexibility of a sports prosthetic leg allows a handicapped person to run. However, traditional algorithms of dynamics computation for a robot system or human skeletal model only consider a rigid-body multi-link system. Recent progress in soft robotics such as piecewise constant strain (PCS) model provides the way to compute dynamics of soft deformation with a low computational cost. We construct a hybrid multi-link system integrating rigid-body and the PCS model. For controlling a humanoid robot with soft links, we implement a dynamics computation with a floating-base and derive the center-of-gravity Jacobian matrix of the hybrid link system. Moreover, we demonstrate a forward dynamics simulation of a humanoid robot with prosthetic legs.

Our research team has been developing biped robots based on the nature of passive dynamics. We aim to both investigate the effect of wobbling mass and apply the findings to biped robots to achieve high-performance running. We used an elastically supported wobbling mass in the trunk of biped robots because humans utilize their elastic organs in the upper body and arms to improve running performance. To investigate the characteristics and mechanisms of passive dynamics of the wobbling mass focusing on the vertical ground reaction force, we used a simple model equipped with an elastically supported wobbling mass and analytically derived periodic solutions. The normal mode analysis of the obtained solutions explained the mechanism under which periodic solutions are achieved, and analytic solutions showed similar vertical locomotion to human running and suggested the mechanism under which high-performance locomotion is achieved from the viewpoint of the ground reaction force and energy efficiency. Based on these solutions, we conducted experiments using a prototype and a biped robot equipped with wobbling mass. Experimental results showed similar characteristics in the vertical movement to the simple model and human running. These findings will help improve our understanding of biped running and contribute to producing human-like running of biped robots.

In this paper we propose a novel flexible and optimization-free controller for standard torque-controlled manipulator for Robotic-Assisted Minimally Invasive Surgery. A novel method has been developed to model the constraint introduced by the laparoscopic tool, i.e. the remote center of motion, exploiting closed chain manipulators theory, and the final controller was synthesized considering the effects the constraint produces at a dynamic level. A set of simulations has been performed in a trajectory tracking task to validate the performances of the proposed controller. Performances have been also tested in a real experimental scenario with a KUKA LWR 4+ with 7 degrees of freedom endowed with a laparoscopic-like tool. Results show the effectiveness of the proposed controller and its capability of modifying the trajectory in order to preserve the RCM constraint.

A dynamic model of a soft fibre-reinforced fluidic elastomer is presented and experimentally verified, which can be used for model-based controller design. Due to the inherent visco-(hyper)elastic characteristics and nonlinear time-dependent behaviour of soft fluidic elastomer robots, analytic dynamic modelling is challenging. The fibre reinforced noninflatable soft fluidic elastomer robot used in this paper can produce both planar and spatial movements. Dynamic equations are developed for both cases. Parameters, related to the viscoelastic behaviour of the robot during elongation and bending motion, are identified experimentally and incorporated into our model. The modified dynamic model is then validated in experiments comparing the time responses of the physical robot with the corresponding outputs of the simulation model. The results validate the accuracy of the proposed dynamic model.

Hand Gesture Recognition (HGR) has application in Human Machine Interfaces (HMIs), to control robots, games, and machines. Here we demonstrate a soft-matter multi-layer printed electronic circuit, that can be used to detect the human gesture without the need for physical contact, except for unlocking the system. The film is able to detect touch and proximity of the hand at various nodes, and thus is able to construct an image of the hand. This is performed through fabrication of a thin, stretchable, high resolution, and multilayer circuit, that embeds a mesh of capacitive sensors for human gesture detection, through proximity sensing. The film is fabricated by superposing 6 layers: 3 insulating polymer layers, two EGaIn liquid metal layers patterned in shape of a mesh, and one flexible printed circuit board (FPCB) layer. The fabrication technique absed on spray deposition of EGaIn liquid metal is fast, low-cost, and easy, and can be used for scalable fabrication of large area multi-node sensing surfaces, with application in Robotic e-skins, and general Human Machine Interfaces. The system is trained with five gestures, including dynamic gestures. We demonstrate classification of these gestures using a support vector machine (SVM) based algorithm.

While a human is tracking a moving object to prepare for later grasping, we naturally change our hand pose to generate optimal pre-grasp to avoid post-grasp adjustment. Robot hand controllers need dynamic pre-grasp planning capability, so they are not limited in dynamic tracking and catching tasks. To fill this gap, we explore the feasibility of using a two-stage optimization method to enable dynamic pre-grasp planning of individual fingers while tracking a moving object to ensure a later successful grasp. The first stage adopts multi-agent pursuit to partition the search space on the object surface. The method allows each finger to consider its immediate surroundings in a local view instead of globally determining the best location for all fingers. The search space for each finger is dramatically reduced since sensible alternatives are the ones left after pruning. Each finger goal location acts independently yet coordinates with others to achieve the goal of covering the object. In the second stage, four different goal point movement strategies are presented to impact the finger goal location in their respective search space to demonstrate the ability to facilitate different needs of the task and requirements of the designer. Dynamic finger goal adaption is obtained by iteratively updating these two stages. The approach is consistent in different scenarios for the object.

Tactile sensing is critical for humans to perform everyday tasks. While significant progress has been made in analyzing object grasping from vision, it remains unclear how we can utilize tactile sensing to reason about and model the dynamics of hand-object interactions. In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We propose a framework aiming at predicting the 3d locations of both the hand and the object purely from the touch data by combining a predictive model and a contrastive learning module. This framework can reason about the interaction patterns from the tactile data, hallucinate the changes in the environment, esti-mate the uncertainty of the prediction, and generalize to unseen objects. We also provide detailed ablation studies regarding different system designs as well as visualizations of the predicted trajectories. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.

In the context of autonomous vehicles, one of the most crucial tasks is to estimate the risk of the undertaken action. While navigating in complex urban environments, the Bayesian occupancy grid is one of the most popular types of maps, where the information of occupancy is stored as the probability of collision. Although widely used, this kind of representation is not well suited for risk assessment: because of its discrete nature, the probability of collision becomes dependent on the tessellation size. Therefore, risk assessments on Bayesian occupancy grids cannot yield risks with meaningful physical units. In this article, we propose an alternative framework called Dynamic Lambda-Field that is able to assess generic physical risks in dynamic environments without being dependent on the tessellation size. Using our framework, we are able to plan safe trajectories where the risk function can be adjusted depending on the scenario. We validate our approach with quantitative experiments, showing the convergence speed of the grid and that the framework is suitable for real-world scenarios.