ICRA 2021 Papers

Simulating soft robots in cluttered environments remains an open problem due to the challenge of capturing complex dynamics and interactions with the environment. Furthermore, fast simulation is desired for quickly exploring robot behaviors in the context of motion planning. In this paper, we examine a particular class of inflated-beam soft growing robots called "vine robots," and present a dynamics simulator that captures general behaviors, handles robot-object interactions, and runs faster than real time. The simulator framework uses a simplified multi-link, rigid-body model with contact constraints. To bridge the sim-to-real gap, we develop methods for fitting model parameters based on video data of a robot in motion and in contact with an environment. We provide examples of simulations, including several with fit parameters, to show the qualitative and quantitative agreement between simulated and real behaviors. Our work demonstrates the capabilities of this high-speed dynamics simulator and its potential for use in the control of soft robots.

This paper proposes a drive-through recharging strategy for a quadrotor. First, a ground charging station is constructed with a portable charging wire, then with two conductive hooks connected to the battery, the quadrotor is flying through the station. Finally, after connecting the hooks to the charging wire, the quadrotor can be recharged without landing or stopping, similar to a drive-through. During the charging, the quadrotor is still able to accomplish local monitoring assignments. The main challenge of this work is to ensure that the hooks successfully connect to the charging wire. Though visual-based techniques appear promising to handle this problem, the detection of a thin charging wire is challenging especially during the flight. Besides, the conductive hooks underneath the quadrotor are out of the camera view, which brings more challenges to the problem. Therefore, in this paper, the aiming problem is directed into the position constraint control problem of the quadrotor. Specifically, a virtual aiming box is considered near the charging wire center, then a control design is proposed to constrain the position of the quadrotor to ensure the hooks position always within the virtual aiming box. Moreover, to enhance the system robustness against the outdoor environment, a wind disturbance estimator is proposed in the control design. Both theoretical analysis and experimental validation are carried out to demonstrate the effectiveness of the proposed strategy.

EMG-driven musculoskeletal model has been broadly used to detect human intention in rehabilitation robots. This approach computes muscle-tendon force and translates it to the joint kinematics. However, the muscle-tendon parameters of the musculoskeletal model are difficult to measure in vivo and varied across subjects. In this study, a direct collocation (DC) method is proposed to optimize the subject-specific parameters in a wrist musculoskeletal model. The resultant optimized parameters are used to estimate the wrist flexion/extension motion. The estimation performance is compared with the parameters optimized by the genetic algorithm. Experiment results show that the DC methods have a similar performance compared with GA, in which the mean correlation are 0.96 and 0.93 for the genetic algorithm and DC method respectively. But the direction collocation method requires less optimization time.

Robotic exoskeletons are exciting technologies for augmenting human mobility. However, designing such a device for seamless integration with the human user and to assist human movement still is a major challenge. This paper aims at developing a novel data-driven solution framework based on reinforcement learning (RL), without first modeling the human-robot dynamics, to provide optimal and adaptive personalized torque assistance for reducing human efforts during walking. Our automatic personalization solution framework includes the assistive torque profile with two control timing parameters (peak and offset timings), the least square policy iteration (LSPI) for learning the parameter tuning policy, and a cost function based on a transferred work ratio. The proposed controller was successfully validated on a healthy human subject to assist unilateral hip extension in walking. The results showed that the optimal and adaptive RL controller as a new approach was feasible for tuning assistive torque profile of the hip exoskeleton that coordinated with human actions and reduced activation level of hip extensor muscle in human.

This paper considers a scenario where a robot and a human operator share the same workspace, and the robot is able to both carry out autonomous tasks and physically interact with the human in order to achieve common goals. In this context, both intentional and accidental contacts between human and robot might occur due to the complexity of tasks and environment, to the uncertainty of human behavior, and to the typical lack of awareness of each other actions. Here, a two stage strategy based on Recurrent Neural Networks (RNNs) is designed to detect intentional and accidental contacts: the occurrence of a contact with the human is detected at the first stage, while the classification between intentional and accidental is performed at the second stage. An admittance control strategy or an evasive action is then performed by the robot, respectively. The approach also works in the case the robot simultaneously interacts with the human and the environment, where the interaction wrench of the latter is modeled via Gaussian Mixture Models (GMMs). Control Barrier Functions (CBFs) are included, at the control level, to guarantee the satisfaction of robot and task constraints while performing the proper interaction strategy. The approach has been validated on a real setup composed of a Kinova Jaco2 robot.

Motion planning for robotic manipulation makes heavy use of quasistatic models, but these same models have not yet proven useful for simulation. This is because in many multi-contact situations, the quasistatic models do not describe a unique next state for the system. A planner is able to use these models optimistically (checking only for feasibility of a motion), but simulation requires more.In this work, we enable quasistatic models to uniquely determine contact forces by modeling actuated robots as impedances instead of prescribed motions. Using this model with a well-known convex relaxation for Coulomb friction, time-stepping of quasistatic models can be formulated as a convex Quadratic Program (QP). This convex relaxation does admit mild non-physical behavior between relatively-sliding objects, but through simulations of various complexity, we show that the proposed quasistatic time-stepping scheme generates mostly physically-realistic behaviors, and scales well with the complexity of the simulated systems.

Reliable operation in inclement weather is essential to the deployment of safe autonomous vehicles (AVs). Robustness and reliability can be achieved by fusing data from the standard AV sensor suite (i.e., lidars, cameras) with weather robust sensors, such as millimetre-wavelength radar. Critically, accurate sensor data fusion requires knowledge of the rigidbody transform between sensor pairs, which can be determined through the process of extrinsic calibration. A number of extrinsic calibration algorithms have been designed for 2D (planar) radar sensors—however, recently-developed, low-cost 3D millimetre-wavelength radars are set to displace their 2D counterparts in many applications. In this paper, we present a continuous-time 3D radar-to-camera extrinsic calibration algorithm that utilizes radar velocity measurements and, unlike the majority of existing techniques, does not require specialized radar retroreflectors to be present in the environment. We derive the observability properties of our formulation and demonstrate the efficacy of our algorithm through synthetic and real-world experiments.

Autonomous robotic suturing has the potential to improve surgery outcomes by leveraging accuracy, repeatability, and consistency compared to manual operations. However, achieving full autonomy in complex surgical environments is not practical and human supervision is required to guarantee safety. In this paper, we develop a confidence-based supervised autonomous suturing method to perform robotic suturing tasks via both Smart Tissue Autonomous Robot (STAR) and surgeon collaboratively with the highest possible degree of autonomy. Via the proposed method, STAR performs autonomous suturing when highly confident and otherwise asks the operator for possible assistance in suture positioning adjustments. We evaluate the accuracy of our proposed control method via robotic suturing tests on synthetic vaginal cuff tissues and compare them to the results of vaginal cuff closures performed by an experienced surgeon. Our test results indicate that by using the proposed confidence-based method, STAR can predict the success of pure autonomous suture placement with an accuracy of 94.74%. Moreover, via an additional 25% human intervention, STAR can achieve a 98.1% suture placement accuracy compared to an 85.4% accuracy of completely autonomous robotic suturing. Finally, our experiment results indicate that STAR using the proposed method achieves 1.6 times better consistency in suture spacing and 1.8 times better consistency in suture bite sizes than the manual results.

Teleoperation—i.e., controlling a robot with human motion—proves promising in enabling a humanoid robot to move as dynamically as a human. But how to map human motion to a humanoid robot matters because a human and a humanoid robot rarely have identical topologies and dimensions. This work presents an experimental study that utilizes reaction tests to compare joint space and task space mappings for dynamic teleoperation of an anthropomorphic robotic arm that possesses human-level dynamic motion capabilities. The experimental results suggest that the robot achieved similar and, in some cases, human-level dynamic performances with both mappings for the six participating human subjects. All subjects became proficient at teleoperating the robot with both mappings after practice, despite that the subjects and the robot differed in size and link length ratio and that the teleoperation required the subjects to move unintuitively. Yet, most subjects developed their teleoperation proficiencies more quickly with task space mapping than with joint space mapping after similar amounts of practice. This study also indicates the potential values of three-dimensional task space mapping, a teleoperation training simulator, and force feedback to the human pilot for intuitive and dynamic teleoperation of a humanoid robot’s arms.

Most of the existing road detection methods are either single-modal based, e.g., based on LiDAR or camera, or multi-modal based with LiDAR-camera fusion. The algorithms are designed for a specific data type, and cannot cope with input data changes. In addition, the LiDAR-camera based methods can only work in day time with enough light. In this paper, we develop a novel LiDAR-camera fusion strategy, which combines the LiDAR point clouds and the camera images in a cascaded way. The proposed network has two working modes, the single-modal mode with LiDAR point clouds only and the multimodal mode with both LiDAR and camera data, so it can be used in all day scenes. The whole network consists of three parts: 1) LiDAR segmentation module, which segments road points in the LiDAR’s imagery view. 2) Sparse-to-dense module, which upsamples the sparse LiDAR feature maps to dense road detection results. 3) LiDAR-camera fusion module, which fuses the dense LiDAR feature maps with the dense camera images to obtain accurate road estimations. Experiments on the KITTI-Road dataset show that the proposed cascaded LiDAR-camera fusion network can obtain very competitive road detection performance, with a MaxF value of 96.38%, and achieve the state-of-the-art in the single-modal mode among all LiDAR-only methods.

This work considers the control of underactuated bipedal walking, and a novel capturability-based control framework is presented. Compared with traditional approaches, the presented control method does not rely on the use of the Poincaré map, which may take significant computational cost. Firstly, a new definition of stable walking is presented, and a novel foot-placement based control method is proposed. Then, a controller design method is presented based on this control method. For the controller design, the foot placement adjustment is achieved by updating the virtual constraints using a heuristic method, and an improved virtual constraint control method is proposed to enforce the virtual constraints. Finally, the effectiveness of the presented control framework is illustrated on a five-link underactuated planar biped by numerical simulations.

The surface electromyography (sEMG) signal-based human-machine interface (HMI) has been widely used for various scenarios of physical human-robot interaction. However, current HMIs based on bipolar myoelectric sensors are hindered by the limitations of global sEMG features, which are prone to variability and delay. In this letter, we define a HMI that takes advantage of the underlying neural information of spinal module activations from bipolar sEMG signals, inspired by recent findings of neural codes. Firstly, the spinal module activations are identified by the spiking trains of the muscle synergies extracted from bipolar sEMG signals. Secondly, we extract the information encoded in both firing rates and spike timings of the spinal module activation in a population coding manner, which follows the information encoding principle of neurons. Thirdly, we map the series of spinal module activations into gait phases, locomotion modes, joint moment and human identity in order to experimentally reveal the physiological information contained in the spinal module activations. The contained information and the benefit of our design are demonstrated and experimentally explained by the presented results and comparisons with the traditionally used global sEMG features. The proposed bipolar myoelectric sensor-enabled human-machine interface could contribute to various scenarios of physical human-robot interaction.

Panoptic segmentation is the recently introduced task that tackles semantic segmentation and instance segmentation jointly [18]. In this paper, we present an extension of SemanticKITTI [1], a large-scale dataset providing dense point-wise semantic labels for all sequences of the KITTI Odometry Benchmark [10]. This extension enables training and evaluation of LiDAR-based panoptic segmentation. We provide the data and discuss the processing steps needed to enrich a given semantic annotation with temporally consistent instance information, i.e., instance information that supplements the semantic labels and identifies the same instance over sequences of LiDAR point clouds. Additionally, we present two strong baselines that combine state-of-the-art LiDAR-based semantic segmentation approaches with a state-of-the-art detector enriching the segmentation with instance information and that allow other researchers to compare their approaches against. We believe that our extension of SemanticKITTI with strong baselines enables the creation of novel algorithms for LiDAR-based panoptic segmentation as much as it has for the original semantic segmentation and semantic scene completion tasks. Data, code, and an online evaluation service using a hidden test set are publicly available at http://semantic-kitti.org.

In this article, a novel 3D printed knee model for detecting possible errors in the planning and treatment of total knee replacement surgery is presented. This method is a first step towards automation of prosthesis placement. Thanks to the mathematically computed four-bar mechanism, it is now possible to emulate the implant insertion before the operation. This model allows the surgeon to fit the prosthesis to the patient’s knee and to verify if the mounted position is optimal. This process can be repeated until the ideal position is found. Exact copies of the bones are made from the CT images of the patient. These resin copies of the real bones are placed in the motion model, reproducing the real patient’s knee flexion. The model shows the implantation result with respect to the operation plan and the patient’s kinematics. The experiment carried out on a patient’s model according to the standard implantation shows a lift-off and sliding effect of the femoral component outside the joint area. However, this phenomenon is easily resolved if the femoral component is implanted about half a centimeter lateral to the knee’s coronal axis. Until now, no medical research had questioned the lateral positioning of the prosthesis. This knee model provides an understanding of the important biomechanical parameters for total knee replacement surgery. This model will have a significant contribution to secure the surgery’s success, increase patient satisfaction and reduce the overall number of revisions.

Due to the high positioning accuracy and relatively low prices, SCARA robots are widely used in industrial fields. The objective of this paper is to propose a hand-eye self-calibration algorithm for SCARA robots which could consider both accuracy and computational cost. The previous global optimal hand-eye calibration algorithms based on branch-and-bound (BnB) optimization is limited by their expensive computational cost. The speed of these algorithms depends on the volume of the search space to a large extent, which is the main concern in this paper. Instead of searching over the 3-dimensional parameter space corresponding to the rotation component of hand-eye pose, a new 2-dimensional search space is defined by separating and coupling some calibration parameters by means of the special structure of SCARA robots, which have 4 degrees of freedom (DoFs) including three translation DoFs and only one rotation DoF. The simulation and real experiments show the similar accuracy but much faster speed of the proposed algorithm compared with previous optimal algorithms based on BnB.

This work addresses the problem of 6-DoF pose estimation under heavy occlusion. While previous work demonstrates reasonable results in unoccluded situations, robust and efficient pose estimation is still challenging in heavily occluded and low-texture scenarios which are ubiquitous in many applications. To this end, we propose a novel end-to-end deep neural network model recovering object poses from depth measurements. The proposed model enforces pairwise consistency of 3D geometric features by applying spectral convolutions on a pairwise compatibility graph. We achieve comparable accuracy as the state-of-the-art graph matching solver while being much faster. Our approach outperforms state-of-the-art 6-DoF pose estimation methods on LineMOD and Occlusion LineMOD and runs in reasonable time (~5.9 Hz). We additionally verify this method on a synthetic dataset with large affine changes.

Proposing grasp poses for novel objects is an essential component for any robot manipulation task. Planning six degrees of freedom (DoF) grasps with a single camera, however, is challenging due to the complex object shape, incomplete object information, and sensor noise. In this paper, we present a 6-DoF contrastive grasp proposal network (CGPN) to infer 6-DoF grasps from a single-view depth image. First, an image encoder is used to extract the feature map from the input depth image, after which 3-DoF grasp regions are proposed from the feature map with a rotated region proposal network. Feature vectors that within the proposed grasp regions are then extracted and refined to 6-DoF grasps. The proposed model is trained offline with synthetic grasp data. To improve the robustness in reality and bridge the simulation-to-real gap, we further introduce a contrastive learning module and variant image processing techniques during the training. CGPN can locate collision-free grasps of an object using a single-view depth image within 0.5 second. Experiments on a physical robot further demonstrate the effectiveness of the algorithm. The experimental videos are available at [1].

With the advent of powerful, light-weight 3D LiDARs, they have become the hearth of many navigation and SLAM algorithms on various autonomous systems. Pointcloud registration methods working with unstructured pointclouds such as ICP are often computationally expensive or require a good initial guess. Furthermore, 3D feature-based registration methods have never quite reached the robustness of 2D methods in visual SLAM. With the continuously increasing resolution of LiDAR range images, these 2D methods not only become applicable but should exploit the illumination-independent modalities that come with it, such as depth and intensity. In visual SLAM, deep learned 2D features and descriptors perform exceptionally well compared to traditional methods. In this publication, we use a state-of-the-art 2D feature network as a basis for 3D3L, exploiting both intensity and depth of LiDAR range images to extract powerful 3D features. Our results show that these keypoints and descriptors extracted from LiDAR scan images outperform state-of-the-art on different benchmark metrics and allow for robust scan-to-scan alignment as well as global localization.

A novel control scheme for biped robots to manipulate the ZMP three-dimensionally apart from the actual ground profile is presented. It is shown that the linear inverted-pendulum-like dynamics with this scheme can represent a wider class of movements including variation of the body height. Moreover, this can also represent the motion in aerial phase. Based on this, the foot-guided controller proposed by the authors is enhanced to enable the robots to locomote on highly uneven terrains and also to seamlessly transition between walking and running without pre-planning the overall motion reference. The controller guarantees the capturability at landing and defines the motion by a time-variant state feedback, which is analytically derived from a model predictive optimization. It is verified through some computer simulations.