ICRA 2022 Papers

Object classification is a key element that enables effective decision-making in many autonomous systems. A more sophisticated system may also utilize the probability distribution over the classes instead of basing its decision only on the most likely class. This paper introduces new performance metrics: the absolute class error (ACE), expectation of absolute class error (EACE) and variance of absolute class error (VACE) for evaluating the accuracy of such probabilities. We test this metric using different neural network architectures and datasets. Furthermore, we present a new task-based neural network for object classification and compare its performance with a typical probabilistic classification model to show the improvement with threshold-based probabilistic decision-making.

The humanoid robot that can transform itself into a form according to its purpose requires whole-body motions with complex contact state transitions such as recovery from a fall and transition to the target form. To make the robot behavior in simulations closer to that in the real world for planning complex target trajectories, we need a platform that can measure the body stiffness during the motion and verify its application without being damaged by repeated motions that are prone to tipping over. In this study, we propose a small, inexpensive, and robust humanoid-vehicle transformer platform with redundant sensors and a low rigidity multi degree-of-freedom body and observe the effects of body deflection and internal forces during whole-body posture transition. By comparing the results obtained from experiments in several environments with different friction and from the simulator using a rigid body model, we were able to verify the influence of body flexibility on whole-body motion and the relationship between deflection and wrench observed by redundant sensors and movement failure.

As the study of humanoid robots becomes a world-wide interdisciplinary research field, the demand for a cost-effective bipedal robot system capable of dynamic behaviors is growing exponentially. This paper presents a miniature bipedal robot named Bipedal Robot Unit with Compliance Enhanced (BRUCE). Each leg of BRUCE has five degrees of freedom (DoFs), which includes a spherical hip joint, a knee joint, and an ankle joint. To lower the leg inertia, a cable-driven differential pulley system and a linkage mechanism are applied to the hip and ankle joints, respectively. With the proposed design, BRUCE is able to achieve a similar range of motion to a human's lower body. The proprioceptive actuation and contact sensing further prepare BRUCE for interactions with unstructured environments. For real-time control of dynamic motions, a convex formulation for model hierarchy predictive control (MHPC) is introduced. MHPC plans with whole-body dynamics in the near horizon and simplified dynamics in the long horizon to benefit from both model accuracy and computational efficiency. A series of experiments were conducted to evaluate the overall system performance including hip joint analysis, walking, push recovery, and vertical jumping.

This paper presents a long-range magnetic actuated and guided endoscope for uniport video-assisted thoracic surgery (VATS). In VATS, the incision is quite narrow and part of the chest wall may be very thick. So, the magnetic endoscope system is required to produce sufficient attractive force at a considerable distance with a compact dimension. In this paper, a magnetic endoscope system is developed to meet the aforementioned clinical demands. In the system, both the internal and external units consist of two cylindrical magnets at both ends and a semi-cylindrical magnet in the middle. Coupled with the magnetic field from the external unit, the internal endoscope can achieve anchoring, tilting, panning, and translating to provide the desired view for the surgeon. The rotation of the endoscope is dynamically modeled by combining magnetic theory and coordinate transformation. The prototype is made with a boundary box of 10×14×56 mm, which can be inserted through the narrow incision in VATS. In the experiment, the developed models of anchoring, tilting, and panning were verified. The magnet configuration in the system can achieve a static anchoring distance of 95 mm and exhibits enhancement in attractive force compared with other designs.

Collaborative robots need to work closely and safely with users while being fast and strong. Fulfilling both these needs simultaneously presents a significant challenge, if not a roadblock, for conventional geared motor technology. Magnetorheological (MR) actuation is an alternative technology that has the potential to exhibit both safety and speed at the same time in a compact and cost-effective envelope. MR actuation has demonstrated great potential for low-DOF mechatronic devices in close collaboration with humans such as exoskeletons and flight control systems but its potential for high-DOF collaborative robots remains widely unexplored. This paper presents the design and experimental validation of a 6 DOF manipulator prototype actuated by semi-delocalized MR clutches. The manipulator is designed with the objective of matching or exceeding the performance requirements of today's cobots in order to verify the potential of MR actuation for such applications. Experimental results show that the prototype has a mass in motion of 5.3 kg and can move a 4.5 kg payload at 1 m/s in a range of 0.885 m. Force bandwidth is above 50 Hz and backdriving forces less than 10% of the joints maximum torque, assuring excellent dynamic performance. Furthermore, the manipulator prototype is shown to be inherently safe and impact-tolerant. In all, results suggest that semi-delocalized MR actuation is a promising solution for high performance cobots although future work is needed for the MR technology to reach full-maturity in robotics.

In this paper we present a compositing image synthesis method that generates RGB canvases with well aligned segmentation maps and sparse depth maps, coupled with an in-painting network that transforms the RGB canvases into high quality RGB images and the sparse depth maps into pixel-wise dense depth maps. We benchmark our method in terms of structural alignment and image quality, showing an increase in mIoU over SOTA by 3.7 percentage points and a highly competitive FID. Furthermore, we analyse the quality of the generated data as training data for semantic segmentation and depth completion, and show that our approach is more suited for this purpose than other methods.

Vision-and-language navigation (VLN) has been an important task in the field of Robotics and Computer Vision. However, most existing vision-and-language navigation models only use features extracted from RGB observation as input, while robots can utilize depth sensors in the real world. Existing research has also shown that simply adding a depth stream to neural models could only provide a marginal improvement to the performance of the VLN task. Therefore, in our work, we develop a novel method for the VLN task using semantic map observations built from RGB-D input. We use vision-pretraining to efficiently encode the semantic map with CNN and scene query attention network by answering queries about semantic information of specific regions of a scene. The proposed method could be used with a simple model and does not require large-scale vision-language transformer pretraining, bringing a more than 10% increase in the success rate compared with a baseline model. When used together with the Speaker-Follower training technique, it achieves a success rate of 58 % on the test set for the R2R dataset in single-run setting, outperforming the previous RGB-D method and most existing RGB-only models that do not use large-scale vision-language transformers pretraining.

Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving. Approaches to monocular 3D perception including detection and tracking, however, often yield inferior performance when compared to LiDAR-based techniques. Through systematic analysis, we identified that per-object depth estimation accuracy is a major factor bounding the performance. Motivated by this observation, we propose a multi-level fusion method that combines different representations (RGB and pseudo-LiDAR) and temporal information across multiple frames for objects (tracklets) to enhance per-object depth estimation. Our proposed fusion method achieves the state-of-the-art performance of per-object depth estimation on the Waymo Open Dataset, the KITTI detection dataset, and the KITTI MOT dataset. We further demonstrate that by simply replacing estimated depth with fusion-enhanced depth, we can achieve significant improvements in monocular 3D perception tasks, including detection and tracking.

This paper describes rubble recognition using a depth image sensor and an automatic rubble crushing system using a construction machine for automatic rubble crushing at a building demolition site. Depth Distribution Split Labeling (DDSL) is proposed to recognize irregularly shaped rubble using depth images and to identify the largest rubble in the workspace. In DDSL, we focused on the fact that the depth of the contact area of adjacent rubble is lower, and labeling by dividing the depth makes it possible to identify the rubble. The automatic rubble crushing system enables to avoid excessive pushing and to grasp objects based on the cylinder driving force calculated from the hydraulic cylinder internal pressure of the construction machine. Through evaluation experiments simulating the crushing environment, it was confirmed that the proposed system can identify and automatically crush rubble.

Exploiting internal spatial geometric constraints of sparse LiDARs is beneficial to depth completion, however, has been not explored well. This paper proposes an efficient method to learn geometry-aware embedding, which encodes the local and global geometric structure information from 3D points, e.g., scene layout, object's sizes and shapes, to guide dense depth estimation. Specifically, we utilize the dynamic graph representation to model generalized geometric relationship from irregular point clouds in a flexible and efficient manner. Further, we joint this embedding and corresponded RGB appearance information to infer missing depths of the scene with well structure-preserved details. The key to our method is to integrate implicit 3D geometric representation into a 2D learning architecture, which leads to a better trade-off between the performance and efficiency. Extensive experiments demonstrate that the proposed method outperforms previous works and could reconstruct fine depths with crisp boundaries in regions that are over-smoothed by them. The ablation study gives more insights into our method that could achieve significant gains with a simple design, while having better generalization capability and stability. The code is available at https://github.com/Wenchao-Du/GAENet.

Autonomous vehicle-based traffic smoothing con-trollers are often not transferred to real-world use due to challenges in calibrating many-agent traffic simulators. We show a pipeline to sidestep such calibration issues by collecting trajectory data and learning controllers directly from trajectory data that are then deployed zero-shot onto the highway. We construct a dataset of 772.3 kilometers of recorded drives on the I–24. We then construct a simple simulator using the recorded drives as the lead vehicle in front of a simulated platoon consisting of one autonomous vehicle and five human followers. Using policy-gradient methods with an asymmetric critic to learn the controller, we show that we are able to improve average MPG by 11% in simulation on congested trajectories. We deploy this controller to a mixed platoon of 4 autonomous Toyota RAV-4's and 7 human drivers in a validation experiment and demonstrate that the expected time-gap of the controller is maintained in the real world test. Finally, we release the driving dataset [1], the simulator, and the trained controller at https://github.com/nathanlct/trajectory-training-icra.

Increasing the performance of tactile sensing in robots enables versatile, in-hand manipulation. Vision-based tactile sensors have been widely used as rich tactile feedback has been shown to be correlated with increased performance in manipulation tasks. Existing tactile sensor solutions with high resolution have limitations that include low accuracy, expensive components, or lack of scalability. In this paper, an inexpensive, scalable, and compact tactile sensor with high-resolution surface deformation modeling for surface reconstruction of the 3D sensor surface is presented. By observing the contact surface with a fisheye camera, it is shown that the surface deformation can be estimated in real-time (1.8 ms) using deep convolutional neural networks. This sensor in its design and sensing abilities represents a significant step toward better object in-hand localization, classification, and surface estimation all enabled by calibrated, high-resolution shape reconstruction.

We propose a demonstration-efficient strategy to compress a computationally expensive Model Predictive Controller (MPC) into a more computationally efficient representation based on a deep neural network and Imitation Learning (IL). By generating a Robust Tube variant (RTMPC) of the MPC and leveraging properties from the tube, we introduce a data augmentation method that enables high demonstration-efficiency, capable of compensating the distribution shifts typically encountered in IL. Our approach opens the possibility of zero-shot transfer from a single demonstration collected in a nominal domain, such as a simulation or a robot in a lab/controlled environment, to a domain with bounded model errors/perturbations. Numerical and experimental evaluations performed on a trajectory tracking MPC for a multirotor show that our method outperforms strategies commonly employed in IL, such as DAgger and Domain Randomization, in terms of demonstration-efficiency and robustness to perturbations unseen during training.

Autonomous robotic surgery requires deliberation, i.e. the ability to plan and execute a task adapting to uncer-tain and dynamic environments. Uncertainty in the surgical domain is mainly related to the partial pre-operative knowledge about patient-specific anatomical properties. In this paper, we introduce a logic-based framework for surgical tasks with deliberative functions of monitoring and learning. The DE-liberative Framework for Robot-Assisted Surgery (DEFRAS) estimates a pre-operative patient-specific plan, and executes it while continuously measuring the applied force obtained from a biomechanical pre-operative model. Monitoring module compares this model with the actual situation reconstructed from sensors. In case of significant mismatch, the learning module is invoked to update the model, thus improving the estimate of the exerted force. DEFRAS is validated both in simulated and real environment with da Vinci Research Kit executing soft tissue retraction. Compared with state-of-the-art related works, the success rate of the task is improved while minimizing the interaction with the tissue to prevent unintentional damage.

Smart weeding systems to perform plant-specific operations can contribute to the sustainability of agriculture and the environment. Despite monumental advances in autonomous robotic technologies for precision weed management in recent years, work on under-canopy weeding in fields is yet to be realized. A prerequisite of such systems is reliable detection and classification of weeds to avoid mistakenly spraying and, thus, damaging the surrounding plants. Real-time multi-class weed identification enables species-specific treatment of weeds and significantly reduces the amount of herbicide use. Here, our first contribution is the first adequately large realistic image dataset AIWeeds (one/multiple kinds of weeds in one image), a library of about 10,000 annotated images of flax and the 14 most common weeds in fields and gardens taken from 20 different locations in North Dakota, California, and Central China. Second, we provide a full pipeline from model training with maximum efficiency to deploying the TensorRT-optimized model onto a single board computer. Based on AIWeeds and the pipeline, we present a baseline for classification performance using five benchmark CNN models. Among them, MobileNetV2, with both the shortest inference time and lowest memory consumption, is the qualified candidate for real-time applications. Finally, we deploy MobileNetV2 onto our own compact autonomous robot SAMBot for real-time weed detection. The 90% test accuracy realized in previously unseen scenes in flax fields (with a row spacing of 0.2-0.3 m), with crops and weeds, distortion, blur, and shadows, is a milestone towards precision weed control in the real world. We have publicly released the dataset and code to generate the results at https://github.com/StructuresComp/Multi-class-Weed-Classification.

How can a robot navigate successfully in rich and diverse environments, indoors or outdoors, along office corridors or trails on the grassland, on the flat ground or the staircase? To this end, this work aims to address three challenges: (i) complex visual observations, (ii) partial observability of local visual sensing, and (iii) multimodal robot behaviors conditioned on both the local environment and the global navigation objective. We propose to train a neural network (NN) controller for local navigation via imitation learning. To tackle complex visual observations, we extract multi-scale spatial representations through CNNs. To tackle partial observability, we aggregate multi-scale spatial information over time and encode it in LSTMs. To learn multimodal behaviors, we use a separate memory module for each behavior mode. Importantly, we integrate the multiple neural network modules into a unified controller that achieves robust performance for visual navigation in complex, partially observable environments. We implemented the controller on the quadrupedal Spot robot and evaluated it on three challenging tasks: adversarial pedestrian avoidance, blind-spot obstacle avoidance, and elevator riding. The experiments show that the proposed NN architecture significantly improves navigation performance.

Open challenges for deep reinforcement learning systems are their adaptivity to changing environments and their efficiency w.r.t. computational resources and data. In the application of learning lane-change behavior for autonomous driving, the number of required transitions imposes a bottleneck, since test drivers cannot perform an arbitrary amount of lane changes in the real world. In the off-policy setting, additional information on solving the task can be gained by observing actions from others. While in the classical RL setup this knowledge remains unused, we use other drivers as surrogates to learn the agent's value function more efficiently. We propose Surrogate Q-learning that deals with the aforementioned problems and reduces the required driving time drastically. We further propose an efficient implementation based on a permutation equivariant deep neural network architecture of the Q-function to estimate action-values for a variable number of vehicles in sensor range. We evaluate our method in the open traffic simulator SUMO and learn well performing driving policies on the real highD dataset.

Automated agricultural applications, i.e., fruit picking require spatial information about crops and, especially, their fruits. In this paper, we present a novel deep reinforcement learning (DRL) approach to determine the next best view for automatic exploration of 3D environments with a robotic arm equipped with an RGB-D camera. We process the obtained images into an octree with labeled regions of interest (ROIs), i.e., fruits. We use this octree to generate 3D observation maps that serve as encoded input to the DRL network. We hereby do not only rely on known information about the environment, but explicitly also represent information about the unknown space to force exploration. Our network takes as input the encoded 3D observation map and the temporal sequence of camera view pose changes, and outputs the most promising camera movement direction. Our experimental results show an improved ROI targeted exploration performance resulting from our learned network in comparison to a state-of-the-art method.

Modern SLAM engines typically rely on high-end sensor rigs and robust algorithms to guarantee the high-quality requirements that self-driving cars and other complex autonomous systems require from 3D point cloud maps. Nonetheless, multiple factors can impact the reconstruction quality and it is not uncommon to end up with generally consistent maps affected by local distortions and artifacts, especially when mapping increasingly larger environments. We tackle the problem of identifying these low-consistency areas in point cloud maps by analyzing the quality of pair-wise point cloud alignments. Rather than relying on geometric consistency analysis or visual inspection, we leverage on deep point networks and formulate the validation as a binary classification problem, allowing us to quickly and effectively identify areas of improvement.