IROS 2022 Papers

The Perspective-n-Point problem aims to estimate the relative pose between a calibrated monocular camera and a known 3D model, by aligning pairs of 2D captured image points to their corresponding 3D points in the model. We suggest an algorithm that runs on weak IoT devices in real-time but still provides provable theoretical guarantees for both running time and correctness. Existing solvers provide only one of these requirements. Our main motivation was to turn the popular DJI's Tello Drone (<90gr, <100) into an autonomous drone that navigates in an indoor environment with no external human/laptop/sensor, by simply attaching a Raspberry PI Zero (<9gr, <100) into an autonomous drone that navigates in an indoor environment with no external human/laptop/sensor, by simply attaching a Raspberry PI Zero (<9gr, <25) to it. This tiny micro-processor takes as input a real-time video from a tiny RGB camera, and runs our PnP solver on-board. Extensive experimental results, open source code, and a demonstration video are included.

Object detection plays an essential role in providing localization, path planning, and decision making capabilities in autonomous navigation systems. However, existing object detection models are trained and tested on a fixed number of known classes. This setting makes the object detection model difficult to generalize well in real-world road scenarios while encountering an unknown object. We address this problem by introducing our framework that handles the issue of unknown object detection and updates the model when unknown object labels are available. Next, our solution includes three major components that address the inherent problems present in the road scene datasets. The novel components are a) Feature-Mix that improves the unknown object detection by widening the gap between known and unknown classes in latent feature space, b) Focal regression loss handling the problem of improving small object detection and intra-class scale variation, and c) Curriculum learning further enhances the detection of small objects. We use Indian Driving Dataset (IDD) and Berkeley Deep Drive (BDD) dataset for evaluation. Our solution provides state-of-the-art performance on open-world evaluation metrics. We hope this work will create new directions for open-world object detection for road scenes, making it more reliable and robust autonomous systems.

The past decade has amply demonstrated the remarkable functionality that can be realized by learning complex input/output relationships. Algorithmically, one of the most important and opaque relationships is that between a problem's structure and an effective solution method. Here, we quantitatively connect the structure of a planning problem to the performance of a given sampling-based motion planning (SBMP) algorithm. We demonstrate that the geometric relationships of motion planning problems can be well captured by graph neural networks (GNNs) to predict SBMP runtime. By using an algorithm portfolio we show that GNN predictions of runtime on particular problems can be leveraged to accelerate online motion planning in both navigation and manipulation tasks. Moreover, the problem-to-runtime map can be inverted to identify subproblems easier to solve by particular SBMPs. We provide a motivating example of how this knowledge may be used to improve integrated task and motion planning on simulated examples. These successes rely on the relational structure of GNNs to capture scalable generalization from low-dimensional navigation tasks to high degree-of-freedom manipulation tasks in 3d environments.

The global grape and wine industry has been considerably impacted by diseases such as downy mildew (DM). Agricultural robots have demonstrated great potential to accurately and rapidly map DM infection for precision applications. Although the robots can autonomously acquire high-resolution images in the vineyard, data processing is mostly performed offline because of network infrastructure and onboard computing power constraints, limiting the use of agricultural robots for field operations. To address this issue, we developed a semantic segmentation model based on the modified DeepLabv3 network for near real time DM segmentation in high resolution images. Compared with state-of-the-art real time semantic segmentation models, the developed one achieved the best efficiency-accuracy balance on the DM dataset using embedded computing devices that can be easily integrated with commercial robotic platforms. DM severity estimation pipeline based on the model also showed a comparable measurement accuracy and statistical power in differentiation of fungicide treatments as the one based on offline semantic segmentation models. This enables the use of robotic perception systems for field operations.

While mobile navigation has been focused on obstacle avoidance, Navigation Among Movable Obstacles (NAMO) via interaction with the environment, is a problem that is still open and challenging. This paper, presents a novel system integration to handle NAMO using visual feedback. In order to explore the capabilities of our introduced system, we explore the solution of the problem via graph-based path planning in a photorealistic simulator (NVIDIA Isaac Sim), in order to identify if the simulation-to-reality (sim2real) problem in robot navigation can be resolved. We consider the case where a wheeled robot navigates in a warehouse, in which movable boxes are common obstacles. We enable online real-time object localization and obstacle movability detection, to either avoid objects or, if it is not possible, to clear them out from the robot planned path by using pushing actions. We firstly test the integrated system in photorealistic environments, and we then validate the method on a real-world mobile wheeled robot (UCL MPPL) and its on-board sensory and computing system.

Underground environments are some of the most challenging for autonomous navigation. The long, featureless corridors, loose and slippery soils, bad illumination and unavailability of global localization make many traditional approaches struggle. In this work, a topological-based navigation system is presented that enables autonomous navigation of a ground robot in mine-like environments relying exclusively on a high-level topological representation of the tunnel network. The topological representation is used to generate high-level topological instructions used by the agent to navigate through corridors and intersections. A convolutional neural network (CNN) is used to detect all the galleries accessible to a robot from its current position. The use of a CNN proves to be a reliable approach to this problem, capable of detecting the galleries correctly in a wide variety of situations. The CNN is also able to detect galleries even in the presence of obstacles, which motivates the development of a reactive navigation system that can effectively exploit the predictions of the gallery detection.

Object Goal Navigation (ObjectNav) task is to navigate an agent to an object category in unseen environments without a pre-built map. In this paper, we solve this task by predicting the distance to the target using semantically-related objects as cues. Based on the estimated distance to the target object, our method directly choose optimal midterm goals that are more likely to have a shorter path to the target. Specifically, based on the learned knowledge, our model takes a bird's-eye view semantic map as input, and estimates the path length from the frontier map cells to the target object. With the estimated distance map, the agent could simultaneously explore the environment and navigate to the target objects based on a simple human-designed strategy. Empirical results in visually realistic simulation environments show that the proposed method outperforms a wide range of baselines on success rate and efficiency. Real-robot experiment also demonstrates that our method generalizes well to the real world.

Autonomously navigating a robot in everyday crowded spaces requires solving complex perception and planning challenges. When using only monocular image sensor data as input, classical two-dimensional planning approaches cannot be used. While images present a significant challenge when it comes to perception and planning, they also allow capturing potentially important details, such as complex geometry, body movement, and other visual cues. In order to successfully solve the navigation task from only images, algorithms must be able to model the scene and its dynamics using only this channel of information. We investigate whether the world model concept, which has shown state-of-the-art results for modeling and learning policies in Atari games as well as promising results in 2D LiDAR-based crowd navigation, can also be applied to the camera-based navigation problem. To this end, we create simulated environments where a robot must navigate past static and moving humans without colliding in order to reach its goal. We find that state-of-the-art methods are able to achieve success in solving the navigation problem, and can generate dream-like predictions of future image-sequences which show consistent geometry and moving persons. We are also able to show that policy performance in our high-fidelity sim2real simulation scenario transfers to the real world by testing the policy on a real robot. We make our simulator, models and experiments available at https://github.com/danieldugas/NavDreams.

In this paper, we present a decentralized control approach based on a Nonlinear Model Predictive Control (NMPC) method that employs barrier certificates for safe navigation of multiple nonholonomic wheeled mobile robots in unknown environments with static and/or dynamic obstacles. This method incorporates a Learned Barrier Function (LBF) into the NMPC design in order to guarantee safe robot navigation, i.e., prevent robot collisions with other robots and the obstacles. We refer to our proposed control approach as NMPC-LBF. Since each robot does not have a priori knowledge about the obstacles and other robots, we use a Deep Neural Network (DeepNN) running in real-time on each robot to learn the Barrier Function (BF) only from the robot's LiDAR and odometry measurements. The DeepNN is trained to learn the BF that separates safe and unsafe regions. We implemented our proposed method on simulated and actual Turtlebot3 Burger robot(s) in different scenarios. The implementation results show the effectiveness of the NMPC-LBF method at ensuring safe navigation of the robots.

Loop closure detection is a key technology for long-term robot navigation in complex environments. In this paper, we present a global descriptor, named Normal Distribution Descriptor (NDD), for 3D point cloud loop closure detection. The descriptor encodes both the probability density score and entropy of a point cloud as the descriptor. We also propose a fast rotation alignment process and use correlation coefficient as the similarity between descriptors. Experimental results show that our approach outperforms the state-of-the-art point cloud descriptors in both accuracy and efficency. The source code is available and can be integrated into existing LiDAR odometry and mapping (LOAM) systems.

When robots operate in real-world off-road environments with unstructured terrains, the ability to adapt their navigational policy is critical for effective and safe navigation. However, off-road terrains introduce several challenges to robot navigation, including dynamic obstacles and terrain uncertainty, leading to inefficient traversal or navigation failures. To address these challenges, we introduce a novel approach for adaptation by negotiation that enables a ground robot to adjust its navigational behaviors through a negotiation process. Our approach first learns prediction models for various navigational policies to function as a terrain-aware joint local controller and planner. Then, through a new negotiation process, our approach learns from various policies' interactions with the environment to agree on the optimal combination of policies in an online fashion to adapt robot navigation to unstructured off-road terrains on the fly. Additionally, we implement a new optimization algorithm that offers the optimal solution for robot negotiation in real-time during execution. Experimental results have validated that our method for adaptation by negotiation outperforms previous methods for robot navigation, especially over unseen and uncertain dynamic terrains.

Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of realworld dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object structure at inference time. We propose a two-stage partsbased training mechanism which allows the object rendering models to generalize well across the configuration space even if the underlying training data has as few as one configuration represented. We demonstrate the efficacy of NARF22 by training configurable renderers on a real-world articulated tool dataset collected via a Fetch mobile manipulation robot. We show the applicability of the model to gradient-based inference methods through a configuration estimation and 6 degree-of-freedom pose refinement task.

Though robot learning is often formulated in terms of discrete-time Markov decision processes (MDPs), physical robots require near-continuous multiscale feedback control. Machines operate on multiple asynchronous sensing modalities, each with different frequencies, e.g., video frames at 30Hz, proprioceptive state at 100Hz, force-torque data at 500Hz, etc. While the classic approach is to batch observations into fixed-time windows then pass them through feed-forward encoders (e.g., with deep networks), we show that there exists a more elegant approach - one that treats policy learning as modeling latent state dynamics in continuous-time. Specifically, we present InFuser, a unified architecture that trains continuous time-policies with Neural Controlled Differential Equations (CDEs). InFuser evolves a single latent state representation over time by (In)tegrating and (Fus)ing multi-sensory observations (arriving at different frequencies), and inferring actions in continuous-time. This enables policies that can react to multi-frequency multi-sensory feedback for truly end-to-end visuomotor control, without discrete-time assumptions. Behavior cloning experiments demonstrate that InFuser learns robust policies for dynamic tasks (e.g., swinging a ball into a cup) notably outperforming several baselines in settings where observations from one sensing modality can arrive at much sparser intervals than others.

Pruning work at high altitude is a dangerous work with a high risk of accidents for human workers. In this research, we propose a multirotor flying robot that is equipped with a wire-suspended device and performs pruning task. We use a saber saw as a cutting tool. If the cutting tool is installed on the body of the multirotor platform, it is difficult for the flying robot to approach the desired work point if there are obstacles such as other branches around the target branch to be pruned. Therefore, in this study, we propose a saber saw suspended from the body of the multirotor platform with two wires. The wire-suspended device is equipped with a saber saw and four ducted fans that produces thrust in any direction on the horizontal plane. This ducted fan system can be used to suppress the swing of the wire-suspended device to make positioning of the saber saw blade to the target point easier, and to improve the efficiency of the cutting and reduce the cutting time by providing a pushing force to the saber saw. As a result, the pruning work could be performed efficiently. Experiments have demonstrated that aerial pruning is possible using the long-reach wire-suspended saber saw.

Tendon-driven continuum robots show promise for use in surgical applications as they can assume complex configurations to navigate along tortuous paths. However, to achieve these complex robot shapes, multiple segments are required as each robot segment can bend only with a single constant curvature. To actuate these additional robot segments, multiple tendons must typically be added on-board the robot, complicating their integration, robot control, and actuation. This work presents a method of achieving two curvatures in a single tendon-driven continuum robot segment through use of a novel magnetic locking mechanism. Thus, the need for additional robot segments and actuating tendons is eliminated. The resulting two curvatures in a single segment are demonstrated in two and three dimensions. Furthermore, the maximum magnetic field required to actuate the locking mechanism for different robot bending angles is experimentally measured to be 6.1 mT. Additionally, the locking mechanism resists unintentional unlocking unless the robot assumes a 0° bending angle and a magnetic field of 18.1 mT is applied, conditions which are not typically reached during routine use of the system. Finally, addressable actuation of two locking mechanisms is achieved, demonstrating the capability of producing multiple curvatures in a single robot segment.

Forest canopies are the biggest habitat for terrestrial life, yet our understanding of environmental processes and biodiversity inside the canopy continues to be limited due to labour and resource intensive data collection. Existing aerial and climbing robots also struggle to access these complex environments, while animals easily navigate them using multiple means of locomotion. Following this insight we present a robot with multimodal mobility obtained by combining aerial and tethered locomotion. After the robot is deployed at the top of the tree, it can descend with the tether and maneuver around leaves and branches with its thrusters. The tether increases robustness and safety and allows for resting as well as emergency retrieval of the system. The aerial locomotion grants the system the ability to move in a conical 3D space constrained by the tether. We modelled the static system and validated the impact of design parameters on it. A simple control architecture for teleoperation is discussed and its performance is analyzed. The proposed multimodal mobility is demonstrated in preliminary outdoor tests, which show how our robot can move within the canopy while continuously monitoring the environment.

Enabling robots to learn from interactions with users is essential to perform service tasks. However, as a robot categorizes objects from multimodal information obtained by its sensors during interactive onsite teaching, the inferred names of unknown objects do not always match the human user's expectation, especially when the robot is introduced to new environments. Confirming the learning results through natural speech interaction with the robot often puts an additional burden on the user who can only listen to the robot to validate the results. Therefore, we propose a human-robot interface to reduce the burden on the user by visualizing the inferred results in mixed reality (MR). In particular, we evaluate the proposed interface on the system usability scale (SUS) and the NASA task load index (NASA-TLX) with three experimental object categorization scenarios based on multimodal latent Dirichlet allocation (MLDA) in which the robot: 1) does not share the inferred results with the user at all, 2) shares the inferred results through speech interaction with the user (baseline), and 3) shares the inferred results with the user through an MR interface (proposed). We show that providing feedback through an MR interface significantly reduces the temporal, physical, and mental burden on the human user compared to speech interaction with the robot.

Domain shifts, such as appearance changes, are a key challenge in real-world applications of activity recognition models, which range from assistive robotics and smart homes to driver observation in intelligent vehicles. For example, while simulations are an excellent way of economical data collection, a Synthetic→Real domain shift leads to > 60% drop in accuracy when recognizing Activities of Daily Living (ADLs). We tackle this challenge and introduce an activity domain generation framework which creates novel ADL appearances (novel domains) from different existing activity modalities (source domains) inferred from video training data. Our frame-work computes human poses, heatmaps of body joints, and optical flow maps and uses them alongside the original RGB videos to learn the essence of source domains in order to generate completely new ADL domains. The model is optimized by maximizing the distance between the existing source appearances and the generated novel appearances while ensuring that the semantics of an activity is preserved through an additional classification loss. While source data multimodality is an important concept in this design, our setup does not rely on multi-sensor setups, (i.e., all source modalities are inferred from a single video only.) The newly created activity domains are then integrated in the training of the ADL classification networks, resulting in models far less susceptible to changes in data distributions. Extensive experiments on the Synthetic→Real benchmark Sims4Action demonstrate the potential of the domain generation paradigm for cross-domain ADL recognition, setting new state-of-the-art results. Our code is publicly available at https://github.com/Zrrr1997/syn2real_DG.

Spatiotemporal calibration of sensors, especially of those which do not share their fields of view, is becoming increasingly important in the fields of autonomous driving and robotics. This paper presents a general sensor calibration method, named Multical, that makes use of multiple planar calibration targets whose poses will be estimated alongside spatiotemporal calibration. Multical exploits continuous-time curves to represent the state of the sensor platform during data collection, and thus is a general framework to calibrate different kinds of sensors and deal with both spatial as well as temporal offsets. Multical includes algorithms to estimate the initial guesses of spatial transformations between sensors, and also the relative poses between calibration targets. Users do not need to provide any extrinsic priors. We apply the proposed calibration approach to both simulated and real-world experiments, and the results demonstrate the high fidelity of the proposed method.