IROS 2022 Papers

The visual attention of navigators is imperative to understand the logic of navigation as well as the surveillance of navigators' status and operation. Current studies are implemented with the help of wearable eye-tracker glasses; yet, the high expenditure demanded by such equipment and service and its limitations on usability have impeded related research from further development. In this letter, the authors propose a framework, which is the first attempt in the maritime domain, to provide a camera-based deep-learning (CaBDeeL) visual attention recognition solution that outperforms the intrusive eye tracker regarding its shortcomings. A wide-angle camera is configured in front of the navigator in the advanced ship-bridge simulator in a way that visual attention reflected by their facial and head movements is captured in the front view. A pair of eye-tracker glasses is used to classify the captured visual attention images, which then form the primary database. During the process of classifying camera-captured images, a convolutional neural network (CNN) is built as an automatic classifier. The CNN is applied to two scenarios, and it shows an overall 95 % accuracy.

Simultaneous localization and mapping (SLAM) is one of the essential techniques and functionalities used by robots to perform autonomous navigation tasks. Inspired by the rodent hippocampus, this paper presents a biologically inspired SLAM system based on a LiDAR sensor using a hippocampal model to build a cognitive map and estimate the robot pose in indoor environments. Based on the biologically inspired models mimicking boundary cells, place cells, and head direction cells, the SLAM system using LiDAR point cloud data is capable of leveraging the self-motion cues from the LiDAR odometry and the boundary cues from the LiDAR boundary cells to build a cognitive map and estimate the robot pose. Experiment results show that with the LiDAR boundary cells the proposed SLAM system greatly outperforms the camera-based brain-inspired method in both simulation and indoor environments, and is competitive with the conventional LiDAR-based SLAM methods.

Precise geolocalization is crucial for unmanned aerial vehicles (UAVs). However, most current deployed UAVs rely on the global navigation satellite systems (GNSS) for geolocalization. In this paper, we propose to use a lightweight visual-inertial system with a 2D georeferenced map to obtain accurate geodetic positions for UAVs. The proposed system firstly integrates a micro inertial measurement unit (MIMU) and a monocular camera to build a visual-inertial odometry (VIO) to consecutively estimate the UAV's motion states and reconstruct the 3D position of the observed visual features in the local world frame. To obtain the geolocation, the visual features tracked by the odometry are further registered to the 2D georeferenced map. While most conventional methods perform image-level aerial image registration, we propose to align the reconstructed 3D points with the map, and then use the registered 3D points to relocalize the vehicle in the geodetic frame, which helps to improve the geolocalization accuracy. Finally, a pose graph is deployed to fuse the geolocation from the point registration and the local navigation result from the visual-inertial odometry (VIO) to obtain smooth and drift-free geolocalization results. We have validated the proposed method by installing the sensors to a UAV body rigidly and have conducted two real-world flights in different environments with unknown initials. The results show that the proposed method can achieve less than 4m position error in flight at about 100m high and less than 9m position error in flight at about 300m high.

Vision-based robotic assembly is a crucial yet challenging task as the interaction with multiple objects requires high levels of precision. In this paper, we propose an integrated 6D robotic system to perceive, grasp, manipulate and assemble blocks with tight tolerances. Aiming to provide an off-the-shelf RGB-only solution, our system is built upon a monocular 6D object pose estimation network trained solely with synthetic images leveraging physically-based rendering. Subsequently, pose-guided 6D transformation along with collision-free assembly is proposed to construct any designed structure with arbitrary initial poses. Our novel 3-axis calibration operation further enhances the precision and robustness by disentangling 6D pose estimation and robotic assembly. Both quantitative and qualitative results demonstrate the effectiveness of our proposed 6D robotic assembly system.

We present a new dataset for 6-DoF pose estimation of known objects, with a focus on robotic manipulation research. We propose a set of toy grocery objects, whose physical instantiations are readily available for purchase and are appropriately sized for robotic grasping and manipulation. We provide 3D scanned textured models of these objects, suitable for generating synthetic training data, as well as RGBD images of the objects in challenging, cluttered scenes exhibiting partial occlusion, extreme lighting variations, multiple instances per image, and a large variety of poses. Using semi-automated RGBD-to-model texture correspondences, the images are annotated with ground truth poses accurate within a few millimeters. We also propose a new pose evaluation metric called ADD-H based on the Hungarian assignment algorithm that is robust to symmetries in object geometry without requiring their explicit enumeration. We share pre-trained pose estimators for all the toy grocery objects, along with their baseline performance on both validation and test sets. We offer this dataset to the community to help connect the efforts of computer vision researchers with the needs of roboticists.11https://github.com/swtyree/hope-dataset

The recent trend towards low reduction gearing in robotic actuation has revitalised the need for high-performance gearing concepts. In this work we propose compact low-reduction cycloidal gearing, that is 3D-printable and combined with off-the-shelf components. This approach presents an enormous potential for high performance-to-cost implementations. After discussing parameter selection and design considerations, we present a prototype that is combined with a low-cost brushless motor to demonstrate its potential. Extensive experimental results demonstrate high performance, including >40Nm torque, low friction and play, and high impact robustness. The results show that the proposed approach can yield viable gearbox designs.

In recent years, there has been a growing interest in robotic manipulation of deformable objects. In order to perform certain tasks, the robot must control the shape of the object while taking care not to apply excessive stresses so as not to deform it irreversibly. This is the case when extracting elasto-plastic objects in strips from an industrial reel. In order to control the mechanical stresses within the object, we propose a vision-based control scheme to minimize tension by regulating the angular velocity of a motorized reel on which they are wound. In this paper, we propose a method, based on a catenary model and visual feedback from a low-cost RGB-D camera, to estimate the tension distribution along a rubber strip. Thus, the control strategy aims to achieve a desired tension value by varying the length of the suspended portion of the manipulated strip. Simulation and experimental results validate the proposed approach for strip-like objects of various dimensions.

This paper proposes a novel 3D single-object tracker to more stably, accurately, and faster track objects, even if they are temporarily missed. Our idea is to utilize spatial-temporal data association to achieve object tracking robustly, and it consists of two main parts. We firstly employ a temporal motion model cross frames to estimate the object's temporal information and update the region of interest(ROI). The advanced detector only focuses on ROI rather than the whole scene to generate the spatial position. Second, we introduce a new pairwise evaluation system to exploit spatial-temporal data association in point clouds. The proposed evaluation system considers detection confidence, orientation offset, and objects distance to more stably achieve object matching. Then, we update the predicted state based on the pairwise spatial-temporal data. Finally, we utilize the previous trajectory to enhance the accuracy of static tracking in the refinement scheme. Experiments on the KITTI and nuScenes tracking datasets demonstrate that our method outperforms other state-of-the-art methods by a large margin (a 10% improvement and 280 FPS on a single NVIDIA 1080Ti GPU). Compared with multi-object tracking, our tracker also has superiority.

Monocular depth estimation has been actively studied in fields such as robot vision, autonomous driving, and 3D scene understanding. Given a sequence of color images, unsupervised learning methods based on the framework of Structure-From-Motion (SfM) simultaneously predict depth and camera relative pose. However, dynamically moving objects in the scene violate the static world assumption, resulting in inaccurate depths of dynamic objects. In this work, we propose a new method to address such dynamic object movements through monocular 3D object detection. Specifically, we first detect 3D objects in the images and build the per-pixel correspondence of the dynamic pixels with the detected object pose while leaving the static pixels corresponding to the rigid background to be modeled with camera motion. In this way, the depth of every pixel can be learned via a meaningful geometry model. Besides, objects are detected as cuboids with absolute scale, which is used to eliminate the scale ambiguity problem inherent in monocular vision. Experiments on the KITTI depth dataset show that our method achieves State-of-The-Art performance for depth estimation. Furthermore, joint training of depth, camera motion and object pose also improves monocular 3D object detection performance. To the best of our knowledge, this is the first work that allows a monocular 3D object detection network to be fine-tuned in a self-supervised manner.

Safe motion planning in robotics requires planning into space which has been verified to be free of obstacles. However, obtaining such environment representations using lidars is challenging by virtue of the sparsity of their depth measurements. We present a learning-aided 3D lidar reconstruction framework that upsamples sparse lidar depth measurements with the aid of overlapping camera images so as to generate denser reconstructions with more definitively free space than can be achieved with the raw lidar measurements alone. We use a neural network with an encoder-decoder structure to predict dense depth images along with depth uncertainty estimates which are fused using a volumetric mapping system. We conduct experiments on real-world outdoor datasets captured using a handheld sensing device and a legged robot. Using input data from a 16-beam lidar mapping a building network, our experiments showed that the amount of estimated free space was increased by more than 40% with our approach. We also show that our approach trained on a synthetic dataset generalises well to real-world outdoor scenes without additional fine-tuning. Finally, we demonstrate how motion planning tasks can benefit from these denser reconstructions.

Three-dimensional (3D) human pose estimation is one of the most basic tasks for human-interacting robots. Especially in weightless environments such as the International Space Station (ISS), wherein objects may move with a higher degree of freedom compared to on the ground, a camera with a wider field of view (FOV) is crucial in improving the probability of capturing surrounding humans. To this end, we propose a learning-based 3D human pose estimation (3D-HPE) using a fisheye camera targeted at weightless environments. One impediment is that 3D-HPE trained on existing datasets are incapable of addressing the adverse effects of strong fisheye distortion and weightlessness as existing human detection and pose estimation datasets are recorded on the ground using typical rectilinear cameras. To overcome this difficulty, we integrate virtual camera projection to perform a detected human-centered undistortion from fisheye to rectilinear images. We also include upside-down augmentation during training to improve the performance toward weightlessness. Our results show that these two techniques successfully mitigate the adverse effects of weightlessness and fisheye distortion.

As an abstract representation of the environment structure, a topological map has advantageous properties for path-planning and navigation. Here we proposed an online topological mapping method, 360ST-Mapping, using omnidirectional vision. The 360° field-of-view allows the agent to obtain consistent observation and incrementally extract topological environment information. Moreover, we leverage semantic infor-mation to guide topological place recognition, further improving performance. The topological map possessing semantic infor-mation has the potential to support semantics-related advanced tasks. After integrating the topological mapping module into the omnidirectional visual SLAM system, we conducted extensive experiments in several large-scale indoor scenes and validated the method's effectiveness.

Person detection is a crucial task for mobile robots navigating in human-populated environments. LiDAR sensors are promising for this task, thanks to their accurate depth measurements and large field of view. Two types of LiDAR sensors exist: the 2D LiDAR sensors, which scan a single plane, and the 3D LiDAR sensors, which scan multiple planes, thus forming a volume. How do they compare for the task of person detection? To answer this, we conduct a series of exper-iments, using the public, large-scale JackRabbot dataset and the state-of-the-art 2D and 3D LiDAR-based person detectors (DR-SPAAM and CenterPoint respectively). Our experiments include multiple aspects, ranging from the basic performance and speed comparison, to more detailed analysis on localization accuracy and robustness against distance and scene clutter. The insights from these experiments highlight the strengths and weaknesses of 2D and 3D LiDAR sensors as sources for person detection, and are especially valuable for designing mobile robots that will operate in close proximity to surrounding humans (e.g. service or social robot).