IROS 2022 Papers

A kitchen assistant needs to operate human-scale objects, such as cabinets and ovens, in unmapped environments with dynamic obstacles. Autonomous interactions in such environments require integrating dexterous manipulation and fluid mobility. While mobile manipulators in different form factors provide an extended workspace, their real-world adoption has been limited. Executing a high-level task for general objects requires a perceptual understanding of the object as well as adaptive whole-body control among dynamic obstacles. In this paper, we propose a two-stage architecture for autonomous interaction with large articulated objects in unknown environments. The first stage, object-centric planner, only focuses on the object to provide an action-conditional sequence of states for manipulation using RGB-D data. The second stage, agent-centric planner, formulates the whole-body motion control as an optimal control problem that ensures safe tracking of the generated plan, even in scenes with moving obstacles. We show that the proposed pipeline can handle complex static and dynamic kitchen settings for both wheel-based and legged mobile manipulators. Compared to other agent-centric planners, our proposed planner achieves a higher success rate and a lower execution time. We also perform hardware tests on a legged mobile manipulator to interact with various articulated objects in a kitchen. For additional material, please check: www.pair.toronto.edularticulated-mm/.

Reliability of SLAM systems is considered one of the critical requirements in modern autonomous systems. This directed the efforts to developing many state-of-the-art systems, creating challenging datasets, and introducing rigorous metrics to measure SLAM performance. However, the link between datasets and performance in the robustness/resilience context has rarely been explored. In order to fill this void, characterization of the operating conditions of SLAM systems is essential in order to provide an environment for quantitative measurement of robustness and resilience. In this paper, we argue that for proper evaluation of SLAM performance, the characterization of SLAM datasets serves as a critical first step. The study starts by reviewing previous efforts for quantitative characterization of SLAM datasets. Then, the problem of perturbation characterization is discussed and the linkage to SLAM robustness/resilience is established. After that, we pro-pose a novel, generic and extendable framework for quantitative analysis and comparison of SLAM datasets. Additionally, a description of different characterization parameters is provided. Finally, we demonstrate the application of our framework by presenting the characterization results of three SLAM datasets: KITTI, EuroC-MAV, and TUM-VI highlighting the level of insights achieved by the proposed framework.

In this study, we present an application of the Piece-wise Constant Strain (PCS) model to a flexible deformation analysis of a sports prosthesis leg. Dynamic motion analysis of an athlete wearing a sports prosthesis is important to clarify a relationship between the prosthesis characteristics and the performance of an athlete, which contributes to training of an athlete or design of the prosthesis. However, there are few studies on modeling of the three-dimensional deformation of the sports prosthesis. In soft robotics, the PCS model was proposed for calculating a flexible deformation of a beam or rod structure with a low computational cost. We employ the PCS model to calculate the flexible deformation of the prosthesis, assuming that its structure can be discretized into a finite number of segments. Moreover, we propose an estimation method of the prosthesis stiffness using optical motion capture data and calculating the semi-definite programming.

Feature extraction or localization based on the fiducial marker could fail due to motion blur in real-world robotic applications. To solve this problem, a lightweight generative adversarial network, named Ghost-DeblurGAN, for real-time motion deblurring is developed in this paper. Furthermore, on account that there is no existing deblurring benchmark for such task, a new large-scale dataset, York-Tag, is proposed that provides pairs of sharp/blurred images containing fiducial markers. With the proposed model trained and tested on YorkTag, it is demonstrated that when applied along with fiducial marker systems to motion-blurred images, Ghost-DeblurGAN improves the marker detection significantly. The datasets and codes used in this paper are available at: https://github.com/York-SDCNLab/Ghost-DeblurGAN.

Rescue robots that search around on debris during natural disasters require high mobility to overcome various shaped materials scattered in the environment. Our previous study developed a new tracked mechanism called Mono-wheel Track, an elastic track driven by a single wheel, having a high capability to get over obstacles. In designing the MW - Track, the track stiffness is an essential factor-the flexible track can adapt to the geometry of the obstacles, but the flexibility prevents grousers from anchoring to the environment steadily. If the track has different localized stiffnesses, both the adaptability and the stability might be archived. In this study, we developed an “anisotropic-stiffness track,” exhibiting different stiffness depending on the bending side, and investigated its deformation characteristics and the effects on mobility. The basic deformation characteristics of the track were confirmed by load tests. The effects on mobility were evaluated by step-climbing tests, ditch-crossing tests, and traction measuring with a mobile robot.

This work presents a motion capture-driven locomotion controller for quadrupedal robots that replicates the non-periodic footsteps and subtle body movement of animal motions. We adopt a nonlinear model predictive control (NMPC) formulation that generates optimal base trajectories and stepping locations. By optimizing both footholds and base trajectories, our controller effectively tracks retargeted animal motions with natural body movements and highly irregular strides. We demonstrate our approach with prerecorded animal motion capture data. In simulation and hardware experiments, our motion controller enables quadrupedal robots to robustly reproduce fundamental characteristics of a target animal motion regardless of the significant morphological disparity.

Reinforcement Learning (RL) for robotic applications can benefit from a warm-start where the agent is initialized with a pretrained behavioral policy. However, when transitioning to RL updates, degradation in performance can occur, which may compromise the robot's safety. This degradation, which constitutes an inability to properly utilize the pretrained policy, is attributed to extrapolation error in the value function, a result of high values being assigned to Out-Of-Distribution actions not present in the behavioral policy's data. We investigate why the magnitude of degradation varies across policies and why the policy fails to quickly return to behavioral performance. We present visual confirmation of our analysis and draw comparisons to the Offline RL setting which suffers from similar difficulties. We propose a novel method, Confidence Constrained Learning (CCL) for Warm-Start RL, that reduces degradation by balancing between the policy gradient and constrained learning according to a confidence measure of the Q-values. For the constrained learning component we propose a novel objective, Positive Q-value Distance (CCL-PQD). We investigate a variety of constraint-based methods that aim to overcome the degradation, and find they constitute solutions for a multi-objective optimization problem between maximimal performance and miniminal degradation. Our results demonstrate that hyperparameter tuning for CCL-PQD produces solutions on the Pareto Front of this multi-objective problem, allowing the user to balance between performance and tolerable compromises to the robot's safety.

Optimization-based robot control strategies often rely on first-order dynamics approximation methods, as in iLQR. Using second-order approximations of the dynamics is expensive due to the costly second-order partial derivatives of the dynamics with respect to the state and control. Current approaches for calculating these derivatives typically use automatic differentiation (AD) and chain-rule accumulation or finite-difference. In this paper, for the first time, we present analytical expressions for the second-order partial derivatives of inverse dynamics for open-chain rigid-body systems with floating base and multi-DoF joints. A new extension of spatial vector algebra is proposed that enables the analysis. A recursive algorithm with complexity of \mathcal{O}(Nd^{2})\mathcal{O}(Nd^{2}) is also provided where N is the number of bodies and d is the depth of the kinematic tree. A comparison with AD in CasADi shows speedups of 1.5-3 x for serial kinematic trees with N > 5, and a C++ implementation shows runtimes of \approx \mathbf{5 1} \mu \mathrm{s}\approx \mathbf{5 1} \mu \mathrm{s} for a quadruped.

This paper discusses the tele-operation system for multiple mobile manipulators. If a single person could freely tele-operate multiple mobile manipulators simultaneously, it would be a great step toward the goal of “avatar-symbiotic society” allowing people to live beyond the constraints of their bodies, space, and time. At present, however, such a tele-operation system has not been developed. Therefore, we built a prototype system to tele-operate two mobile manipulators and conducted a subject experiment on a pick-and-place task to investigate the tele-operator's task performance, workload and gaze behavior. By analyzing these results, we obtained guidelines to design tele-operation system for multiple robots.

Randomization is currently a widely used approach in Sim2Real transfer for data-driven learning algorithms in robotics. Still, most Sim2Real studies report results for a specific randomization technique and often on a highly customized robotic system, making it difficult to evaluate different randomization approaches systematically. To address this problem, we define an easy-to-reproduce experimental setup for a robotic reach-and-balance manipulator task, which can serve as a benchmark for comparison. We compare four randomization strategies with three randomized parameters both in simulation and on a real robot. Our results show that more randomization helps in Sim2Real transfer, yet it can also harm the ability of the algorithm to find a good policy in simulation. Fully randomized simulations and fine-tuning show differentiated results and translate better to the real robot than the other approaches tested.

Many operations performed by work class Remotely Operated Vehicles (ROVs) require the manipulation of heavy loads. An example is the manipulation and grouting of armour stones. A way to increase the working capabilities of the ROV is to introduce cables among the set of actuators. The cable lengths and tensions are controlled by winches placed on the vehicle. Being similar to a cable-driven parallel robot (CDPR), the resultant system inherits some advantages such as the possibility to generate large forces over a large workspace and the possibility to use CDPR techniques to estimate the pose of the ROV. This paper proposes a complete control architecture for the Hybrid Cable-Thruster actuated ROV (HCT-ROV) and analyzes, in computer simulations, the performances of such a system while it performs real world operations, such as heavy lifting and hovering in presence of water current.

We present an open source motion planning framework for ROS, which uses constraint and optimization based task space control to generate trajectories for the whole body of mobile manipulators. Motion goals are defined as constraints which are enforced on task space functions. They map the controllable degrees of freedom of a system onto custom task spaces, which can, but do not have to be, the Cartesian space. We use this expressive tool from motion control to pre-compute trajectories in order to utilize the fact that most robots offer controllers to follow such trajectories. As a result, our framework only requires a kinematic model of the robot to control it. In addition, we extend the constraint-based motion control approach with linear MPC to explicitly optimize for velocity, acceleration and jerk simultaneously, which allows us to enforce constraints on all derivatives in both joint and task space at the same time. As a result, we can reuse predefined motion goals on any robot without modifications. Our framework was tested on four different robots to show its generality.

We propose a new approach to control the shape of deformable objects with robots. Specifically, we consider a fixed-length elastic linear object lying on a 2D workspace. Our main idea is to encode the object's deformation behavior in an offline constant Jacobian matrix. To derive this Jacobian, we use geometric deformation modeling and combine recent work from the fields of deformable object control and multirobot systems. Based on this Jacobian, we then propose a robotic control law that is capable of driving a set of shape features on the object toward prescribed values. Our contribution relative to existing approaches is that at run-time we do not need to measure the full shape of the object or to estimate/simulate a deformation model. This simplification is achieved thanks to having abstracted the deformation behavior as an offline model. We illustrate the proposed approach in simulation and in experiments with real deformable linear objects.

Integrating different representations from complementary sensing modalities is crucial for robust scene interpretation in autonomous driving. While deep learning architectures that fuse vision and range data for 2D object detection have thrived in recent years, the corresponding modalities can degrade in adverse weather or lighting conditions, ultimately leading to a drop in performance. Although domain adaptation methods attempt to bridge the domain gap between source and target domains, they do not readily extend to heterogeneous data distributions. In this work, we propose an unsupervised domain adaptation framework, which adapts a 2D object detector for RGB and LiDAR sensors to one or more target domains featuring adverse weather conditions. Our proposed approach consists of three components. First, a data augmentation scheme that simulates weather distortions is devised to add domain confusion and prevent overfitting on the source data. Second, to promote cross-domain foreground object alignment, we leverage the complementary features of multiple modalities through a multi-scale entropy-weighted domain discriminator. Finally, we use carefully designed pretext tasks to learn a more robust representation of the target domain data. Experiments performed on the DENSE dataset show that our method can substantially alleviate the domain gap under the single-target domain adaptation setting and the less explored yet more general multi-target domain adaptation setting.

This paper proposes an underwater target perception framework to comprehensively explore target information in underwater scenes, to improve the work efficiency and safety of underwater operations. This framework adopts a layered processing mechanism including water column imaging, constant false alarm rate detection (CFAR) detection, and local feature analysis, to accurately distinguish between false targets, static targets, and dynamic targets in the underwater scene, and obtain the motion trajectory of dynamic targets. The experiment is designed to simulate the underwater operation scene, and the results prove the effectiveness of the proposed framework.

This paper presents an optimal motion planning framework to generate versatile energy-optimal quadrupedal jumping motions automatically (e.g., flips, spin). The jumping motions via the centroidal dynamics are formulated as a 12-dimensional black-box optimization problem subject to the robot kino-dynamic constraints. Gradient-based approaches offer great success in addressing trajectory optimization (TO), yet, prior knowledge (e.g., reference motion, contact schedule) is required and results in sub-optimal solutions. The new proposed framework first employed a heuristics-based optimization method to avoid these problems. Moreover, a prioritization fitness function is created for heuristics-based algorithms in robot ground reaction force (GRF) planning, enhancing convergence and searching performance considerably. Since heuristics-based algorithms often require significant time, motions are planned offline and stored as a pre-motion library. A selector is designed to automatically choose motions with user-specified or perception information as input. The proposed framework has been successfully validated only with a simple continuously tracking PD controller in an open-source Mini-Cheetah by several challenging jumping motions, including jumping over a window-shaped obstacle with 30 cm height and left-flipping over a rectangle obstacle with 27 cm height. (Video*)

Virtual links and virtual joints can be appended to the kinematic chain of a robot arm to assist in modelling and control of certain tasks. Activities such as spray painting, sand blasting, or scanning with a laser or camera can be enhanced by modelling the fluid stream, light beam, or field of view using a virtual link. Virtual joints can be used to allow movement in semi-redundant degrees of freedom of the task space. This can can be exploited to optimize the control of the real robot. A prudent choice is to minimize the effort required by the manipulator to execute the task. This often requires the inversion of the inertia matrix. However, virtual links have no inertia so the inverse does not exist. This paper first explores methods of adding virtual mass or modifying the inertia matrix to allow inversion and the consequences. Then an optimal control problem is proposed that minimizes kinetic energy in the real manipulator and maximizes use of the virtual joints. In doing so, we only need the real inertia matrix which is always invertible. The method is validated in a case study for high pressure water blasting. It is shown to reduce the dynamic torque norm compared to a minimum velocity controller.

Robot navigation in human crowds remains the challenge of understanding human behaviors in different scenarios. We present an approach for interactive and human-friendly crowd navigation in complex static environments. The planner models the online interactions among the robot, humans, and the static environment based on game theory. It recurrently expands and optimizes the estimated trajectories for the robot and neighboring agents and provides human-friendly navigation commands. We use various indicators to evaluate the social awareness of the planners and show that our method outperforms existing approaches in success rate to reach the goals and compatibility with humans while maintaining low navigation times. The planner is successfully deployed on a real-world quadrupedal robot, demonstrating safe and interactive crowd navigation with real-time performance.

This paper proposes a variable impedance control architecture to facilitate eye surgery training in a dual-user haptic system. In this system, an expert surgeon (the trainer) and a novice surgeon (the trainee) collaborate on a surgical procedure using their own haptic devices. The mechanical impedance parameters of the trainer's haptic device remain constant during the operation, whereas those of the trainee vary with his/her proficiency level. The trainee's relative proficiency might be objectively quantified in real-time based on position error between the trainer and the trainee. The proposed architecture enables the trainer to intervene in the training process as needed to ensure the trainee is following the right course of action and to avoid the trainee's from potential tissue injuries. The stability of the overall nonlinear closed-loop system has been investigated using the input-to-state stability (ISS) criterion. High-gain observer with unknown inputs is considered in this work to estimate the interaction forces. Simulation and experimental results under different scenarios confirm the effectiveness of the proposed control methods.