IROS 2022 Papers

Conditional behavior prediction (CBP) builds up the foundation for a coherent interactive prediction and plan-ning framework that can enable more efficient and less conser-vative maneuvers in interactive scenarios. In CBP task, we train a prediction model approximating the posterior distribution of target agents' future trajectories conditioned on the future trajectory of an assigned ego agent. However, we argue that CBP may provide overly confident anticipation on how the autonomous agent may influence the target agents' behavior. Consequently, it is risky for the planner to query a CBP model. Instead, we should treat the planned trajectory as an intervention and let the model learn the trajectory distribution under intervention. We refer to it as the interventional behavior prediction (IBP) task. Moreover, to properly evaluate an IBP model with offline datasets, we propose a Shapley-value-based metric to verify if the prediction model satisfies the inherent temporal independence of an interventional distribution. We show that the proposed metric can effectively identify a CBP model violating the temporal independence, which plays an important role when establishing IBP benchmarks.

This paper presents a distributed interactive framework to provide high-level position instructions for multi-robot aerial cinematography based on coverage over a hemisphere. The control strategy based on optimization of the coverage functional and geometric relationships over a hemisphere is presented. It enables multiple Unmanned Aerial Vehicles (UAVs) to coordinate their motion while tracking a dynamic (real or virtual) target, and can accommodate high-level human inputs to influence UAV concentration. In this framework, each UAV uses local information combined with exogenous inputs to determine its motion. The two inputs to the system, i.e., the predicted trajectory of the target and user-defined aesthetic preferences, are agnostic to the size of the multi-robot system (MRS). The proposed framework is validated using the PX4 SITL Autopilot simulator in Gazebo, and the scalability of the framework is verified via simulations.

Interactive traffic simulation is crucial to autonomous driving systems by enabling testing for planners in a more scalable and safe way compared to real-world road testing. Existing approaches learn an agent model from large-scale driving data to simulate realistic traffic scenarios, yet it remains an open question to produce consistent and diverse multi-agent interactive behaviors in crowded scenes. In this work, we present InterSim, an interactive traffic simulator for testing autonomous driving planners. Given a test plan trajectory from the ego agent, InterSim reasons about the interaction relations between the agents in the scene and generates realistic trajectories for each environment agent that are consistent with the relations. We train and validate our model on a large-scale interactive driving dataset. Experiment results show that InterSim achieves better simulation realism and reactivity in two simulation tasks compared to a state-of-the-art learning-based traffic simulator.

Many recent works detect 3D objects by several sensor modalities for autonomous driving, where high-resolution cameras and high-line LiDARs are mostly used but relatively expensive. To achieve a balance between overall cost and detection accuracy, many multi-modal fusion techniques have been suggested. In recent years, the fusion of LiDAR and Radar has gained ever-increasing attention, especially 4D Radar, which can adapt to bad weather conditions due to its penetrability. Although features have been fused from multiple sensing modalities, most methods cannot learn interactions from different modalities, which does not make for their best use. Inspired by the self-attention mechanism, we present InterFusion, an interaction-based fusion framework, to fuse 16-line LiDAR with 4D Radar. It aggregates features from two modalities and identifies cross-modal relations between Radar and LiDAR features. In experimental evaluations on the Astyx HiRes 2019 dataset, our method outperformed the baseline by 4.20% mAP in 3D and 10.76% BEV mAP for the car class at the moderate level.

Shared control can help in teleoperated object manipulation by assisting with the execution of the user's intention. To this end, robust and prompt intention estimation is needed, which relies on behavioral observations. Here, an intention estimation framework is presented, which uses natural gaze and motion features to predict the current action and the target object. The system is trained and tested in a simulated environment with pick and place sequences produced in a relatively cluttered scene and with both hands, with possible hand-over to the other hand. Validation is conducted across different users and hands, achieving good accuracy and earliness of prediction. An analysis of the predictive power of single features shows the predominance of the grasping trigger and the gaze features in the early identification of the current action. In the current framework, the same probabilistic model can be used for the two hands working in parallel and independently, while a rule-based model is proposed to identify the resulting bimanual action. Finally, limitations and perspectives of this approach to more complex, full-bimanual manipulations are discussed.

In this study, we present a framework to en-sure seamless transition in humanoid push recovery involving hopping strategy. We propose a method to adaptively change the time constant that integrated the ankle strategy and variable height strategy. This framework excites a hopping motion against a large disturbance, which provides a seamless transition from the variable height to the hopping strategies. We analyze the applicable region of each strategy based on the simplified model. Moreover, we show that the hopping strategy prevents falling through whole-body dynamic simulations.

Robot-assisted arthroscopic surgery is transforming the tradition in orthopaedic surgery. Compliance and stability are essential features that a surgical robot must have for safe physical human-robot interaction (PHRI). Surgical tools attached at the robot end-effector and human-robot interaction will affect the robot dynamics inevitably. This could undermine the utility and stability of the robotic system if the varying robot dynamics are not identified and updated in the robot control law. In this paper, an integrated frame-work for robot impedance control and nonlinear disturbance observer (NDOB)-based compensation of uncertain dynamics is proposed, where the former ensures compliant robot behavior and the latter compensates for dynamic uncertainties when necessary. The combination of impedance controller and NDOB is analyzed theoretically in three scenarios. A complete simulation and experimental studies involving three common conditions are then conducted to evaluate the theoretical analyses. A preliminary ppHRI application on arthroscopic surgery is designed to implement the proposed framework on a robotic surgeonassist system and evaluate its effectiveness experimentally. By integrating impedance controller with NDOB, the proposed framework allows an accurate impedance control when dynamic model inaccuracy and external disturbance exist.

Segmenting object instances is a key task in machine perception, with safety-critical applications in robotics and autonomous driving. We introduce a novel approach to instance segmentation that jointly leverages measurements from multiple sensor modalities, such as cameras and LiDAR. Our method learns to predict embeddings for each pixel or point that give rise to a dense segmentation of the scene. Specifically, our technique applies contrastive learning to points in the scene both across sensor modalities and the temporal domain. We demonstrate that this formulation encourages the models to learn embeddings that are invariant to viewpoint variations and consistent across sensor modalities. We further demonstrate that the embeddings are stable over time as objects move around the scene. This not only provides stable instance masks, but can also provide valuable signals to downstream tasks, such as object tracking. We evaluate our method on the Cityscapes and KITTI-360 datasets. We further conduct a number of ablation studies, demonstrating benefits when applying additional inputs for the contrastive loss.

Wood logs picking is a challenging task to automate. Indeed, logs usually come in cluttered configurations, randomly orientated and overlapping. Recent work on log picking automation usually assume that the logs' pose is known, with little consideration given to the actual perception problem. In this paper, we squarely address the latter, using a data-driven approach. First, we introduce a novel dataset, named TimberSeg 1.0, that is densely annotated, i.e., that includes both bounding boxes and pixel-level mask annotations for logs. This dataset comprises 220 images with 2500 individually segmented logs. Using our dataset, we then compare three neural network architectures on the task of individual logs detection and segmentation; two region-based methods and one attention-based method. Unsurprisingly, our results show that axis-aligned proposals, failing to take into account the directional nature of logs, underperform with 19.03 mAP. A rotation-aware proposal method significantly improve results to 31.83 mAP. More interestingly, a Transformer-based approach, without any inductive bias on rotations, outperformed the two others, achieving a mAP of 57.53 on our dataset. Our use case demonstrates the limitations of region-based approaches for cluttered, elongated objects. It also highlights the potential of attention-based methods on this specific task, as they work directly at the pixel-level. These encouraging results indicate that such a perception system could be used to assist the operators on the short-term, or to fully automate log picking operations in the future.

This paper addresses a path planning problem for a fleet of Unmanned Aerial Vehicles (UAVs) that uses both prior information and online gathered data to efficiently inspect large surfaces such as ship hulls and water tanks. UAVs can detect corrosion patches and other defects on the surface from low-resolution images. If defects are detected, they get closer to the surface for a high-resolution inspection. The prior information provides expected defects locations and is affected by both false positives and false negatives. The mission objective is to prioritize the close-up inspection of defected areas while keeping a reasonable time for the coverage of the entire surface. We propose two solutions to this problem: a coverage algorithm that divides the problem into a set of Traveling Salesman Problems (Part-TSP) and a cooperative frontier approach that introduces frontier utilities to incorporate the prior information (Coop-Frontier). We finally provide extensive simulation results to analyze the performance of these approaches and compare them with alternative solutions. These results suggest that both Part-Tspand Coop-Frontier perform better than the baseline solution. Part-Tsphas the best performance in most cases. However, coop-Frontier is preferable in extreme cases because more robust to inhomogeneous corrosion distribution and imperfect information.

This paper addresses local path re-planning for n-dimensional systems by introducing an informed sampling scheme and cost function to achieve collision avoidance with minimum deviation from an (optimal) nominal path. The proposed informed subset consists of the union of ellipsoids along the specified nominal path, such that the subset efficiently encapsulates all points along the nominal path. The cost function penalizes large deviations from the nominal path, thereby ensuring current safety in the face of potential collisions while retaining most of the overall efficiency of the nominal path. The proposed method is demonstrated on scenarios related to the navigation of autonomous marine crafts.

Semantic segmentation of aerial imagery is an important tool for mapping and earth observation. However, supervised deep learning models for segmentation rely on large amounts of high-quality labelled data, which is labour-intensive and time-consuming to generate. To address this, we propose a new approach for using unmanned aerial vehicles (UAVs) to autonomously collect useful data for model training. We exploit a Bayesian approach to estimate model uncertainty in semantic segmentation. During a mission, the semantic predictions and model uncertainty are used as input for terrain mapping. A key aspect of our pipeline is to link the mapped model uncertainty to a robotic planning objective based on active learning. This enables us to adaptively guide a UAV to gather the most informative terrain images to be labelled by a human for model training. Our experimental evaluation on real-world data shows the benefit of using our informative planning approach in comparison to static coverage paths in terms of maximising model performance and reducing labelling efforts.

In the absence of an absolute positioning system, such as GPS, autonomous vehicles are subject to accumu-lation of positional error which can interfere with reliable performance. Improved navigational accuracy without GPS enables vehicles to achieve a higher degree of autonomy and reliability, both in terms of decision making and safety. This paper details the use of two navigation systems for autonomous agents using magnetic field anomalies to localize themselves within a map; both techniques use the information content in the environment in distinct ways and are aimed at reducing the localization uncertainty. The first method is based on a nonlinear observability metric of the vehicle model, while the second is an information theory based technique which minimizes the expected entropy of the system. These conditions are used to design guidance laws that minimize the localization uncertainty and are verified both in simulation and hardware experiments are presented for the observability approach.

Several template models have been developed to facilitate the analysis of limit-cycles for quadrupedal locomotion. The parameters in the model are usually fixed; however, biology shows that animals change their leg stiffness according to the locomotion velocity, and this adaptability invariably affects the stability of the gait. This paper provides an analysis of the influence of this variable leg stiffness on the stability of different quadrupedal gaits. The analysis exploits a simplified quadrupedal model with compliant legs and shoulder joints represented as torsional springs. This model can reproduce the most common quadrupedal gaits observed in nature. The stability of such emerging gaits is then checked. Afterward, an optimization process is used to search for the system parameters that guarantee maximum gait stability. Our study shows that using the highest feasible leg swing frequency and adopting a leg stiffness that increases with the speed of locomotion noticeably improves the gait stability over a wide range of horizontal velocities while reducing the oscillations of the trunk. This insight can be applied in the design of novel elastic quadrupedal robots, where variable stiffness actuators could be employed to improve the overall locomotion behavior.

Being able to reproduce physical phenomena ranging from light interaction to contact mechanics, simulators are becoming increasingly useful in more and more application domains where real-world interaction or labeled data are difficult to obtain. Despite recent progress, significant human effort is needed to configure simulators to accurately reproduce real-world behavior. We introduce a pipeline that combines inverse rendering with differentiable simulation to create digital twins of real-world articulated mechanisms from depth or RGB videos. Our approach automatically discovers joint types and estimates their kinematic parameters, while the dynamic properties of the overall mechanism are tuned to attain physically accurate simulations. Control policies optimized in our derived simulation transfer successfully back to the original system, as we demonstrate on a simulated system. Further, our approach accurately reconstructs the kinematic tree of an articulated mechanism being manipulated by a robot, and highly nonlinear dynamics of a real-world coupled pendulum mechanism. Website: https://eric-heiden.github.io/video2sim

This paper is interested in formalizing human trajectories in human-robot encounters. Inspired by robot navigation tasks in human-crowded environments, we consider the case where a human and a robot walk towards each other, and where humans have to avoid colliding with the incoming robot. Further, humans may describe different be-haviors, ranging from being in a hurry/minimizing completion time to maximizing safety. We propose a decision tree-based algorithm to extract STL formulae from multi-label data. Our inference algorithm learns STL specifications from data containing multiple classes, where instances can be labelled by one or many classes. We base our evaluation on a dataset of trajectories collected through an online study reproducing human-robot encounters.

This paper deals with the estimation of the shape of a catenary for a negatively buoyant cable, connecting a pair of underwater robots in a robot chain. The new estimation method proposed here is based on the calculation of local tangents thanks to the data acquired from inertial measurement units (IMUs), which are attached to the cable near its ends. This method is compared with a vision-based estimation method that was developed previously. Experiments are conducted, in the air and in a pool, using a motion capture system for ground truth. The results obtained show that the new method significantly improves the estimation of the catenary height. Furthermore, the identification of the cable shape is not affected by the limits of the camera's field of view and by the image projection, resulting in increased accuracy and range, without singularities.

Robot identification is a prolific topic that has a long history with results spanning recent decades. Recent years have witnessed a renew of interest in this problem due in part to a rapid increase in robotic hardware platforms capable of accurate model-based control. The most popular methods exploit the fact that the inverse dynamic model is linear to the dynamic parameters. Because we identify robots with closed-loop procedures, an Instrumental Variable approach called IDIM-IV (Inverse Dynamic Identification Model with Instrumental Variable estimation) that combines the direct and inverse dynamic models to prevent from correlation between errors has been successfully validated. However, IDIM-IV does not guarantee that the direct dynamic model will be well-posed during its iterations because of possible modeling errors. In this paper, we combine physical constraints and IDIM-IV to address this deficiency for IDIM-IV. This new constrained IV approach, called PC-IDIM-IV (Physically Consistent IDIM-IV), consists of two nested iterative algorithms: an outer one that is IDIM-IV and an inner one that accounts for the physical constraints solved by a Gauss-Newton algorithm. Experimental results and comparisons with other methods carried out with the TX40 robot show the feasibility of PC-IDIM-IV.

In this work, we propose IndoLayout, a novel real-time approach for generating high-quality occupancy maps from an RGB image for indoor scenes. Such occupancy maps are often crucial for path-planning and mapping in indoor environments but are often built using only information contained in the ego view. In contrast, our approach also predicts occupancy values beyond immediately visible regions from just a monocular image, leveraging learnt priors from indoor scenes. Hence, our proposed network can produce a hallucinated, amodal scene layout that includes areas occluded in the RGB image, such as a navigable floor behind a desk. Specifically, we propose a novel architecture that uses self-attention and adversarial learning to vastly improve the quality of the predicted layout. We evaluate our model on several photorealistic indoor datasets and outperform previous relevant work on all metrics that measure layout quality, including newly adopted ones. Finally, we demonstrate the effectiveness of our method by showing significant improvements on the PointNav task over similar approaches using IndoLayout. For more details, please refer to the project page: https://indolayout.github.io/.