IROS 2023 Papers

Modern unmanned aerial vehicles (UAVs) with sophisticated mechanics ask for extended online system identification to aid model-based controls in task execution. In addition, UAVs in adverse environmental conditions require a more detailed environmental disturbance understanding. The necessary combination of online system identification, sensor suite self-calibration, and external disturbance analysis to tackle these issues holistically is currently an open issue. Our proposed FUSE-D approach combines these elements based on a system model at the rotor-speed level and a single global pose sensor (e.g., a tracking system like Optitrack). Besides sensor intrinsics and extrinsics, the framework allows estimating the UAV's rotor geometry, mass, moments of inertia, and the rotors' aerodynamic properties, as well as an external force and where it acts on the UAV. The general formulation allows us to extend the approach to an N-rotor (multi-rotor) UAV and classify the type of external disturbance. We perform a detailed non-linear observability analysis for the 43 + 7N states and do a statistically relevant embedded hardware-in-the-loop performance analysis in the realistic simulation environment Gazebo with RotorS.

Many approaches to grasp synthesis optimize analytic quality metrics that measure grasp robustness based on finger placements and local surface geometry. However, generating feasible dexterous grasps by optimizing these metrics is slow, often taking minutes. To address this issue, this paper presents FRoGGeR: a method that quickly generates robust precision grasps using the min-weight metric, a novel, almost-everywhere differentiable approximation of the classical \epsilon\epsilon grasp metric. The min-weight metric is simple and interpretable, provides a reasonable measure of grasp robustness, and admits numerically efficient gradients for smooth optimization. We leverage these properties to rapidly synthesize collision-free robust grasps-typically in less than a second. FRoGGeR can refine the candidate grasps generated by other methods (heuristic, data-driven, etc.) and is compatible with many object representations (SDFs, meshes, etc.), We study FRoGGeR's performance on over 40 objects drawn from the YCB dataset, outperforming a competitive baseline in computation time, feasibility rate of grasp synthesis, and picking success in simulation. We conclude that FRoGGeR is fast: it has a median synthesis time of 0.834s over hundreds of experiments.

Modern computing platforms for robotics applications comprise a set of heterogeneous elements, e.g., multi-core CPUs, embedded GPUs, and FPGAs. FPGAs are reprogrammable hardware devices that allow for fast and energy-efficient computation of many relevant tasks in robotics. ROS is the de-facto programming standard for robotics and decomposes an application into a set of communicating nodes. ReconROS is a previous approach that can map complete ROS nodes into hardware for acceleration. Since ReconROS relies on standard ROS communication layers, exchanging data between hardware-mapped nodes can lead to a performance bottleneck. This paper presents fpgaDDS, a lean data distribution service for hardware-mapped ROS 2 nodes. fpgaDDS relies on a customized and statically generated streaming-based communication architecture. We detail this communication architecture with its components and outline its benefits. We evaluate fpgaDDS on a test example and a larger autonomous vehicle case study. Compared to a ROS 2 application in software, we achieve speedups of up to 13.34 and reduce jitter by two orders of magnitude.

Collaborative localization (CL) has garnered substantial attention in the field of robotics in recent years. Nonetheless, conventional CL algorithms have faced challenges when dealing with practical issues such as spurious sensor data and limited or discontinued observation and communication in real-world settings. This paper proposes a fault-tolerant practical estimated cross-covariance minimum variance update method (FPECMV) designed to tackle these challenges under limited connectivity. The proposed algorithm uses a CNN-based method to evaluate confidence, along with a fault isolation module to identify faults and manage spurious data in real time. The proposed fault isolation module utilizes relative measurement information that randomly occurs, without requiring high observation and communication prerequisites. Notably, the algorithm takes into account correlations among agents to maintain consistency in localization filters and attain accurate localization despite constraints posed by limited connectivity. To evaluate the performance of the proposed algorithm, experiments were conducted in a collaborative multi-robot environment with spurious sensor data and limited connectivity, using both the BULLET simulation and physical mobile robots. The experimental results indicate that the overall localization performance of the proposed algorithm is improved by 21.0% compared to the state of the art. The experiment results demonstrate the effectiveness of our algorithm in localizing group agents in challenging and intricate scenarios with limited connectivity and spurious sensor data.

Visual place recognition is essential for vision-based robot localization and SLAM. Despite the tremendous progress made in recent years, place recognition in changing environments remains challenging. A promising approach to cope with appearance variations is to leverage high-level semantic features like objects or place categories. In this paper, we propose FM-Loc which is a novel image-based localization approach based on Foundation Models. Our approach uses the Large Language Model GPT-3 in combination with the Visual-Language Model CLIP to construct a semantic image descriptor that is robust to severe changes in scene geometry and camera viewpoint. We deploy CLIP to detect objects in an image, GPT-3 to suggest potential room labels based on the detected objects, and CLIP again to propose the most likely location label. The object labels and the scene label constitute an image descriptor that we use to calculate a similarity score between the query and database images. We validate our approach on real-world data that exhibit significant changes in camera viewpoints and object placement between the database and query trajectories. The experimental results demonstrate that our method is applicable to a wide range of indoor scenarios without the need for training or fine-tuning.

Trajectory planning plays a crucial role in autonomous driving systems, as it is tasked to generate feasible trajectories under highly dynamic scenarios within the time constraint. This paper proposes a novel two-stage coarse-to-fine framework for efficient sampling-based trajectory planning. The proposed method is designed to iteratively generate new trajectory samples focused on the low-cost regions in the sampling space. Two trajectory exploration algorithms are well-designed for efficient search in discretized coarse global space and continuous fine local space, respectively. Experimental results on the first-of-its-kind planning benchmark tool CommonRoad show that our method significantly outperforms the baseline methods both in optimality and computational efficiency. Overall, our approach offers a promising solution for efficient and effective trajectory planning in more autonomous vehicle applications.

Trajectory optimization is a powerful tool for robot motion planning and control. State-of-the-art general-purpose nonlinear programming solvers are versatile, handle constraints effectively and provide a high numerical robustness, but they are slow because they do not fully exploit the optimal control problem structure at hand. Existing structure-exploiting solvers are fast, but they often lack techniques to deal with nonlinearity or rely on penalty methods to enforce (equality or inequality) path constraints. This work presents FATROP: a trajectory optimization solver that is fast and benefits from the salient features of general-purpose nonlinear optimization solvers. The speed-up is mainly achieved through the integration of a specialized linear solver, based on a Riccati recursion that is generalized to also support stagewise equality constraints. To demonstrate the algorithm's potential, it is bench-marked on a set of robot problems that are challenging from a numerical perspective, including problems with a minimum-time objective and no-collision constraints. The solver is shown to solve problems for trajectory generation of a quadrotor, a robot manipulator and a truck-trailer problem in a few tens of milliseconds. The algorithm's C++-code implementation accompanies this work as open source software, released under the GNU Lesser General Public License (LGPL). This software framework may encourage and enable the robotics community to use trajectory optimization in more challenging applications.

Due to the advantages of high flexibility, large workspace, and good human-body compatibility, flexible tendon-driven surgical continuum robots have attracted a lot of attention in robot-assisted minimally invasive surgery. However, due to the coupling of the position and angle of the continuum robot, and the easy deformation of the external force, its inverse kinematics solution has always been a challenge. This paper proposes a fast inverse kinematics solver for surgical continuum robots with a variable curvature model. Firstly, the deformation of the continuum robot is analyzed, and a representation method of the variable curvature model is proposed. Next, to solve the inverse kinematics problem when the continuum robot deforms under load, FABRIKv is proposed by improving the Forward And Backward Reaching Inverse Kinematics (FABRIK). During the inverse kinematics solution, the algorithm preserves the real-time nature of FABRIK and corrects for deformation effects caused by the load. Finally, the experiment verifies the rationality and effectiveness of the variable curvature model representation method, as well as the fastness and accuracy of the FARIKv solver.

Bird's Eye View (BEV) representations are tremendously useful for perception-related automated driving tasks. However, generating BEVs from surround-view fisheye camera images is challenging due to the strong distortions introduced by such wide-angle lenses. We take the first step in addressing this challenge and introduce a baseline, F2BEV, to generate discretized BEV height maps and BEV semantic segmentation maps from fisheye images. F2BEV consists of a distortion-aware spatial cross attention module for querying and consolidating spatial information from fisheye image features in a transformer-style architecture followed by a task-specific head. We evaluate single-task and multi-task variants of F2BEV on our synthetic FB-SSEM dataset, all of which generate better BEV height and segmentation maps (in terms of the IoU) than a state-of-the-art BEV generation method operating on undistorted fisheye images. We also demonstrate discretized height map generation from real-world fisheye images using F2BEV. Our dataset is publicly available at https://github.com/volvo-cars/FB-SSEM-dataset

Multi-modal sensing often involves determining correspondences between each domain's signals, which in turn depends on the accurate extrinsic calibration of the sensors. Challengingly, the camera-LIDAR sensor modalities are quite dissimilar and the narrow field of view of most commercial LIDARs means that they observe only a partial view of the camera frustum. We present a framework for extrinsic calibration of a camera and a LIDAR using only a simple off-the-shelf checkerboard. It is designed to operate even when the LIDAR observes a significantly truncated portion of the checkerboard. Current state-of-the-art methods often require bespoke manufactured markers or full observation of the entire checkerboard in both camera and LIDAR data which is prohibitive. By contrast, our novel algorithm directly aligns the LIDAR intensity pattern to the camera-detected checkerboard pattern using our differentiable formulation. The key step for achieving accurate extrinsics estimation is the use of the spatial derivatives provided by the differentiable checkerboard pattern, and jointly optimizing over all views. In our experiments, we achieve calibration accuracy in the order of 2–4 mm and demonstrate a 30% error reduction compared to state-of-the-art approaches. We are able to achieve this improvement while using only partial LIDAR views of the checkerboard that allows for a simpler data capture process. We also demonstrate the generalizability of our approach to different combinations of LIDARs and cameras with varying sparsity patterns and noise levels.

Effective access to relevant environmental changes in large human environments is critical for service robots to perform tasks. Since the position of a dynamic goal such as a human is variable, it will be difficult for the robot to locate him accurately. It is worth noting that humans can obtain information through social software, and deal with daily affairs. The current robots search for targets without considering some implicit information changes, which leads to not searching for the target objects in the end. Therefore, we propose to extract human implicit location change information from group chats dialogues, i.e., watching dialogues in group chats and extracting who, when, and where(3W), to assist robots in finding explicit character targets. Then we propose a dynamic spatiotemporal map(DSTM) to store the change information as knowledge for the robot. When the robot identifies a target person, it needs to follow the changing information in the scene to infer the possible location and probability of the target person, and then develop a search strategy. We deployed our framework on a custom mobile robot and performed instruction navigation tasks in a university building to evaluate our approach. We demonstrate the ability of our framework to collect and use information in a large human social environment.

Dexterous manipulation using an under-actuated hand has been a challenging task due to its non-linear dynamical characteristics. For a linkage-based under-actuated hand designed to be used to grasp and manipulate large, heavy, and rigid objects stably, precision grasping is necessary, which makes the task even more difficult to deal with. While approaches based on external sensors have been introduced throughout the years, to create a robotic hand that can be used for various tasks in unstructured environments, this paper takes the standpoint that control techniques that do not fully depend on utilizing additional sensing elements need to be further developed. This paper applies the hybrid method using analytics models and data-driven-based approaches to analyze internal sensors' data during the operation of the robot and introduces novel data-driven-based techniques to compensate for the limitations of controlling a linkage-based under-actuated hand with a self-locking mechanism. Then, a within-hand object position manipulation framework with proposed methodologies is presented and experimented with to show its effectiveness.

Since modern robots are supposed to work closely together with humans, physical human-robot interaction is gaining importance. One crucial aspect for safe collaboration is a robust collision reaction strategy that is triggered after an unintentional physical contact. In this work, we propose a dynamically-consistent collision reaction controller, where the reactive motion is performed in one particular desired direction in Cartesian space, without disturbing the remaining ones. This results in more intuitive and more predictable behavior of the end-effector. In addition, the proposed reaction control law is independent of contact and internal observer dynamics used for collision detection. The theoretical claims are validated in simulation and experiments. The proposed reaction controller is experimentally compared with a conventional approach for collision reaction. All experiments have been conducted on a torque controlled KUKA LWR IV + lightweight robot.

Implicit communication plays such a crucial role during social exchanges that it must be considered for a good experience in human-robot interaction. This work addresses implicit communication associated with the detection of physical properties, transport, and manipulation of objects. We propose an ecological approach to infer object characteristics from subtle modulations of the natural kinematics occurring during human object manipulation. Similarly, we take inspiration from human strategies to shape robot movements to be communica-tive of the object properties while pursuing the action goals. In a realistic HRI scenario, participants handed over cups - filled with water or empty - to a robotic manipulator that sorted them. We implemented an online classifier to differentiate careful/not careful human movements, associated with the cups' content. We compared our proposed “expressive” controller, which modulates the movements according to the cup filling, against a neutral motion controller. Results show that human kinematics is adjusted during the task, as a function of the cup content, even in reach-to-grasp motion. Moreover, the carefulness during the handover of full cups can be reliably inferred online, well before action completion. Finally, although questionnaires did not reveal explicit preferences from partici-pants, the expressive robot condition improved task efficiency.

Visual pre-training with large-scale real-world data has made great progress in recent years, showing great potential in robot learning with pixel observations. However, the recipes of visual pre-training for robot manipulation tasks are yet to be built. In this paper, we thoroughly investigate the effects of visual pre-training strategies on robot manipulation tasks from three fundamental perspectives: pre-training datasets, model architectures and training methods. Several significant experimental findings are provided that are beneficial for robot learning. Further, we propose a visual pre-training scheme for robot manipulation termed Vi-PRoM, which combines self-supervised learning and supervised learning. Concretely, the former employs contrastive learning to acquire underlying patterns from large-scale unlabeled data, while the latter aims learning visual semantics and temporal dynamics. Extensive experiments on robot manipulations in various simulation environments and the real robot demonstrate the superiority of the proposed scheme. Videos and more details can be found on https://explore-pretrain-robot.github.io.

This work proposes a novel approach to social robot navigation by learning to generate robot controls from a social motion latent space. By leveraging this social motion latent space, the proposed method achieves significant improvements in social navigation metrics such as success rate, navigation time, and trajectory length while producing smoother (less jerk and angular deviations) and more anticipatory trajectories. The superiority of the proposed method is demonstrated through comparison with baseline models in various scenarios. Additionally, the concept of humans' awareness towards the robot is introduced into the social robot navigation framework, showing that incorporating human awareness leads to shorter and smoother trajectories owing to humans' ability to positively interact with the robot.

The study of motion control for the fish-like robots in complex fluid fields is of great importance in improving the performance of underwater vehicles, due to its strong maneuverability, propulsion efficiency, and deceptive visual appearance. In this article, a novel learning-based control framework is first proposed to autonomously explore efficient control policies that are capable of performing motion control tasks in non-quiescent and unknown background flows. First, we utilize a high-fidelity simulation system, named FishGym, to generate various uniform flows. Next, a DRL-based algorithm is incorporated with the FishGym to train the fish-like robot to control its motion to optimally complete a delicately designed task (Approaching Target and Stay) in both quiescent and uniform flow. Then, the obtained control policy together with an online estimator is directly applied to a Path-Following Task. The proposed framework well balances the simulation accuracy and the computational efficiency, which is of crucial importance for effective coupling with the learning algorithm. The simulation results indicate that, via the proposed learning framework, the robot successfully acquired a swimming strategy that can be used to adapt to different background flows and tasks. Furthermore, we also observe some adaptation behavior of the robot, such as rheotaxis, that is similar to the fish in nature, which gains us more insight into the mechanism underlying the adaptation behavior of fish in a complex environment.

For bipedal humanoid robots to successfully operate in the real world, they must be competent at simultaneously executing multiple motion tasks while reacting to unforeseen external disturbances in real-time. We propose Kinodynamic Fabrics as an approach for the specification, solution and simultaneous execution of multiple motion tasks in real-time while being reactive to dynamism in the environment. Kinodynamic Fabrics allows for the specification of prioritized motion tasks as forced spectral semi-sprays and solves for desired robot joint accelerations at real-time frequencies. We evaluate the capabilities of Kinodynamic fabrics on diverse physically-challenging whole-body control tasks with a bipedal humanoid robot both in simulation and in the real-world. Kinodynamic Fabrics outperforms the state-of-the-art Quadratic Program based whole-body controller on a variety of whole-body control tasks on run-time and reactivity metrics in our experiments. Our open-source implementation of Kinodynamic Fabrics as well as robot demonstration videos can be found at this url: https://adubredu.github.io/kinofabs

Robots need to adapt their behaviour while physically interacting with an operator to guarantee safety and provide intuitiveness. Inferring the intentions of the operator is a challenging problem that can be addressed by introducing sensors, in addition to motor encoders. Also, kinematic redundancy can be used to avoid issues such as singularities or mechanical interference, and the redundant coordinates can be controlled freely. In this work, we propose to use the redundant degrees of freedom to infer the intentions of an operator interacting with a backdrivable kinematically redundant parallel robot, without introducing any additional sensors. The proposed approach is based on the fact that, in mechanically backdrivable robots, the operator can control the redundant degrees of freedom, and this can be sensed using solely motor encoders through the solution of the forward kinematics. This approach is implemented to switch between a position controller and a controller that allows the operator to guide the robot freely thanks to gravity compensation. Experiments are carried out to compare this approach with an existing one and show that it improves intuitiveness during interaction by reducing false mode change detections.