IROS 2021 Papers

Manipulation of objects by exploiting their contact with the environment can enhance both the dexterity and payload capability of robotic manipulators. A common way to manipulate heavy objects beyond the payload capability of a robot is to use a sequence of pivoting motions, wherein, an object is moved while some contact points between the object and a support surface are kept fixed. The goal of this paper is to develop an algorithmic approach for automated plan generation for object manipulation with a sequence of pivoting motions. A plan for manipulating a heavy object consists of a sequence of joint angles of the manipulator, the corresponding object poses, as well as the joint torques required to move the object. The constraint of maintaining object contact with the ground during manipulation results in nonlinear constraints in the configuration space of the robot, which is challenging for motion planning algorithms. Exploiting the fact that pivoting motion corresponds to movements in a subgroup of the group of rigid body motions, SE(3), we present a novel task-space based planning approach for computing a motion plan for both the manipulator and the object while satisfying contact constraints. We also combine our motion planning algorithm with a grasping force synthesis algorithm to ensure that friction constraints at the contacts and actuator torque constraints are satisfied. We present simulation results with a dual-armed Baxter robot to demonstrate our approach.

In recent years, there have been many studies on sports robots that can play against humans, including studies on the strategies that sports robots use by taking into account the physical conditions of their opponents. However, there have been few studies on strategies that take into account psychological conditions of the opponents, such as carelessness and habituation. This paper proposes a motion strategy for an air-hockey robot that intentionally coaxes the opponent to learn the robot’s attack sequence and then uses a different attack sequence to catch the opponent off guard. We explicitly model the change in reaction time during serial learning and represent the opponent’s reaction as an evaluation function. By applying this evaluation function to a game tree and selecting the optimal motion, the robot could catch the opponent by surprise. Actual experiments with several subjects confirmed the effectiveness of the proposed method.

Motion planning under uncertainty is one of the main challenges in developing autonomous driving vehicles. In this work, we focus on the uncertainty in sensing and perception, resulted from a limited field of view, occlusions, and sensing range. This problem is often tackled by considering hypothetical hidden objects in occluded areas or beyond the sensing range to guarantee passive safety. However, this may result in conservative planning and expensive computation, particularly when numerous hypothetical objects need to be considered. We propose a reinforcement learning (RL) based solution to manage uncertainty by optimizing for the worst case outcome. This approach is in contrast to traditional RL, where the agents try to maximize the average expected reward. The proposed approach is built on top of the Distributional RL with its policy optimization maximizing the stochastic outcomes’ lower bound. This modification can be applied to a range of RL algorithms. As a proof-of-concept, the approach is applied to two different RL algorithms, Soft Actor-Critic and DQN. The approach is evaluated against two challenging scenarios of pedestrians crossing with occlusion and curved roads with a limited field of view. The algorithm is trained and evaluated using the SUMO traffic simulator. The proposed approach yields much better motion planning behavior compared to conventional RL algorithms and behaves comparably to humans driving style.

Loop closure detection (LCD), which aims to deal with the drift emerging when robots travel around the route, plays a key role in a simultaneous localization and mapping system. Unlike most current methods which focus on seeking an appropriate representation of images, we propose a novel two-stage pipeline dominated by the estimation of spatial geometric relationship. When a query image occurs, we select semantically similar images based on the SuperPoint network and the aggregated selective match kernel in the first stage, and then conduct robust geometric confirmation to verify true loop-closing pairs in the second stage. Based on the potential property of motion field in the LCD scene, a robust feature matching algorithm, termed as motion field consensus with locality preservation (MFC-LP), is proposed. In particular, we exploit the smoothness prior to guide the learning of the motion field for an image pair in a reproducing kernel Hilbert space (RKHS). Meanwhile, to enhance the local relevance of motion vectors, we design a locality preservation mechanism thus making the learned motion field more accurate. Extensive experiments on several publicly available datasets reveal that MFC-LP has a good performance in the general feature matching task and the proposed pipeline outperforms the current state-of-the-art approaches in the LCD task.

Autonomous Valet Parking (AVP) in an under- ground garage is an emerging smart vehicle solution that the community believes to be solvable with close-to-market sensors. Absence of GPS signals and a high degree of self-similarity however render global visual localization in such environments a highly challenging problem. We present a novel underground parking localization method that relies on text recognition in the wild as well as optical character recognition (OCR) to automatically detect parking slot numbers. The detected numbers are then correlated with both geometric as well as semantic information extracted from an offline map of the environment. The resulting measurement model is embedded into a probabilistic Monte-Carlo localization framework. The success of our method is demonstrated on multiple real-world sequences in one of the largest underground parking garages in Shanghai.

Simulation to real (Sim-to-Real) is an attractive approach to construct controllers for robotic tasks that are easier to simulate than to analytically solve. Working Sim-to-Real solutions have been demonstrated for tasks with a clear single objective such as "reach the target". Real world applications, however, often consist of multiple simultaneous objectives such as "reach the target" but "avoid obstacles". A straightforward solution in the context of reinforcement learning (RL) is to combine multiple objectives into a multi-term reward function and train a single monolithic controller. Recently, a hybrid solution based on pre-trained single objective controllers and a switching rule between them was proposed. In this work, we compare these two approaches in the multi-objective setting of a robot manipulator to reach a target while avoiding an obstacle. Our findings show that the training of a hybrid controller is easier and obtains a better success-failure trade-off than a monolithic controller. The controllers trained in simulator were verified by a real set-up.

This paper proposes a novel monocular teach-and-repeat navigation system with the capability of scale awareness, i.e. the absolute distance between observation and goal images. It decomposes the navigation task into a sequence of visual servoing sub-tasks to approach consecutive goal/node images in a topological map. To be specific, a novel hybrid model, named deep steering network is proposed to infer the navigation primitives according to the learned local feature and scale for each visual servoing sub-task. A novel architecture, Scale-Transformer, is developed to estimate the absolute scale between the observation and goal image pair from a set of matched deep representations to assist repeating navigation. The experiments demonstrate that our scale-aware teach-and-repeat method achieves satisfying navigation accuracy, and converges faster than the monocular methods without scale correction given an inaccurate initial pose. The proposed network is integrated into an onboard system deployed on a real robot to achieve real-time navigation in a real environment. A demonstration video can be found online: https://youtu.be/ctlwDaMKnHw

This paper proposes a method to extract the position and pose of vehicles in the 3D world from a single traffic camera. Most previous monocular 3D vehicle detection algorithms focused on cameras on vehicles from the perspective of a driver, and assumed known intrinsic and extrinsic calibration. On the contrary, this paper focuses on the same task using uncalibrated monocular traffic cameras. We observe that the homography between the road plane and the image plane is essential to 3D vehicle detection and the data synthesis for this task, and the homography can be estimated without the camera intrinsics and extrinsics. We conduct 3D vehicle detection by estimating the rotated bounding boxes (r-boxes) in the bird’s eye view (BEV) images generated from inverse perspective mapping. We propose a new regression target called tailed r-box and a dual-view network architecture which boosts the detection accuracy on warped BEV images. Experiments show that the proposed method can generalize to new camera and environment setups despite not seeing imaged from them during training.

Perception is a critical component of high-integrity applications of robotics and autonomous systems, such as self-driving vehicles. In these applications, failure of perception systems may put human life at risk, and a broad adoption of these technologies requires the development of methodologies to guarantee and monitor safe operation. Despite the paramount importance of perception systems, currently there is no formal approach for system-level monitoring. In this work, we propose a mathematical model for runtime monitoring and fault detection and identification in perception systems. Towards this goal, we draw connections with the literature on diagnosability in multiprocessor systems, and generalize it to account for modules with heterogeneous outputs that interact over time. The resulting temporal diagnostic graphs (i) provide a framework to reason over the consistency of perception outputs –across modules and over time– thus enabling fault detection, (ii) allow us to establish formal guarantees on the maximum number of faults that can be uniquely identified in a given perception system, and (iii) enable the design of efficient algorithms for fault identification. We demonstrate our monitoring system, dubbed PerSyS, in realistic simulations using the LGSVL self-driving simulator and the Apollo Auto autonomy software stack, and show that PerSyS is able to detect failures in challenging scenarios (including scenarios that have caused self-driving car accidents in recent years), and is able to correctly identify faults while entailing a minimal computation overhead (< 5 ms on a single-core CPU).

While object detection modules are essential functionalities for any autonomous vehicle, the performance of such modules that are implemented using deep neural networks can be, in many cases, unreliable. In this paper, we develop abstraction-based monitoring as a logical framework for filtering potentially erroneous detection results. Concretely, we consider two types of abstraction, namely data-label abstraction and post-algorithm abstraction. Operated on the training dataset, the construction of data-label abstraction iterates each input, aggregates region-wise information over its associated labels, and stores the vector under a finite history length. Post-algorithm abstraction builds an abstract transformer for the tracking algorithm. Elements being associated together by the abstract transformer can be checked against consistency over their original values. We have implemented the overall framework to a research prototype and validated it using publicly available object detection datasets.

In this paper, we present a planning and control framework for dynamic, whole-body motions for dynamically stable shape-accelerating mobile manipulators. This class of robots are inherently unstable and require careful coordination between the upper and lower body to maintain balance while performing arm motion tasks. Solutions to this problem either use a complex, full-body nonlinear dynamic model of the robot or a highly simplified model of the robot. Here we explore the use of centroidal dynamics which has recently become a popular approach for designing balancing controllers for humanoid robots. We describe a framework where we first solve a trajectory optimization problem offline. We define balancing for a ballbot in terms of the centroidal momentum instead of other approaches like ZMP or angular velocity that are more commonly used. The generated motion is tracked using a PD-PID cascading balancing controller for the body and torque controller for the arms. We demonstrate that this framework is capable of generating dynamic motion plans and control inputs with examples on the CMU ballbot, a single-spherical-wheeled balancing mobile manipulator.

The paper introduces the novel Modular Pipe Climber III with a Three-Output Open Differential (3-OOD) mechanism to eliminate slipping of the tracks due to the changing cross-sections of the pipe. This will be achieved in any orientation of the robot. Previous pipe climbers use three-wheel/track modules, each with an individual driving mechanism to achieve stable traversing. Slipping of tracks is prevalent in such robots when it encounters the pipe turns. Thus, active control of each module’s speed is employed to mitigate the slip, thereby requiring substantial control effort. The proposed pipe climber implements the 3-OOD to address this issue by allowing the robot to mechanically modulate the track speeds as it encounters a turn. The proposed 3-OOD is the first three-output differential to realize the functional abilities of a traditional two-output differential.

Adaptive locomotion of microrobots can be achieved by using a smart polymer such as a hydrogel. For hydrogel-based bilayer helical microrobots, the change of environment such as temperature and pH can result in shape deformation into helical shapes differing from their initial state and hence swimming performance. In this work, we proposed a model for studying the parameters that affect the shape deformation of a hydrogel-based bilayer helical microrobot. Moreover, the dynamics of some examples of responsive helical swimming are compared before and after stimulation.

This paper presents a novel methodology to model and optimize trajectories of a quadrupedal robot with spinal compliance to improve standing jump performance compared to quadrupeds with a rigid spine. We introduce an elastic model for a prismatic robotic spine that is actively preloaded and mechanically lock-enabled at initial and maximum length, and develop a constrained trajectory optimization method to cooptimize the elastic parameters and motion trajectories toward enhanced jumping distance. Results reveal that a less stiff spring is likely to facilitate jumping performance not as a direct propelling source but as a means to unleash more motor power for propelling by trading-off overall energy efficiency. We also visualize the impact of spring coefficients on the overall optimization routine from energetic perspectives to identify the suitable parameter region.

The focus of this paper is (i) to derive a suitable dynamic model for a self-reconfigurable pavement sweeping robot PANTHERA and (ii) to design a robust controller for the same to tackle uncertainties stemming from the reconfiguration process, external disturbances and from actuator saturation. To meet the first objective, an Euler-Lagrangian dynamic model is proposed to incorporate the effects of configuration changes on the system dynamics. Based on this model, the second objective is met via designing a singular perturbation based robust controller which can tackle the aforementioned uncertainties without violating the actuation limits. To circumvent the vulnerability toward actuator saturation, the proposed controller is built on contraction theory, which, compared to a conventional Lyapunov theory based design, allows to improve closed-loop tracking performance without reducing the singular perturbation parameter. Experimental results on the PANTHERA reconfigurable robot validate the effectiveness of the proposed controller over the state of the art.

This paper introduces a planar tensegrity robot that walks passively and cyclically on a gentle downhill, where its gait versatility can be strengthened by applying actuation forces on the connection cables. The novelty of this work is that we design the structure of this passive robot inspired by the rimless wheel, which naturally generates cyclic locomotion. Consequently, its mathematical model is analytically derived based on passive dynamic walking. Besides, the limb support conditions and dynamics effects induced by the collisions can be precisely determined accordingly. Moreover, numerical simulation is performed to show the typical gait pattern, and resonance phenomenon is observed. Finally, a preliminary experimental study is conducted to prove the validity of the mathematical model. The robot we developed and the mathematical model we derived enable further extensions on the gait analysis and model-based control by conveniently adopting efficient passivemimic walking techniques.

Estimating the states of surrounding traffic participants stays at the core of autonomous driving. In this paper, we study a novel setting of this problem: model-free single-object tracking (SOT), which takes the object state in the first frame as input, and jointly solves state estimation and tracking in subsequent frames. The main purpose for this new setting is to break the strong limitation of the popular "detection and tracking" scheme in multi-object tracking. Moreover, we notice that shape completion by overlaying the point clouds, which is a by-product of our proposed task, not only improves the performance of state estimation but also has numerous applications. As no benchmark for this task is available so far, we construct a new dataset LiDAR-SOT and corresponding evaluation protocols based on the Waymo Open dataset [29]. We then propose an optimization-based algorithm called SOTracker involving point cloud registration, vehicle shapes, correspondence, and motion priors. Our quantitative and qualitative results prove the effectiveness of our SOTracker and reveal the challenging cases for SOT in point clouds, including the sparsity of LiDAR data, abrupt motion variation, etc. Finally, we also explore how the proposed task and algorithm may benefit other autonomous driving applications, including simulating LiDAR scans, generating motion data, and annotating optical flow. The code and protocols for our benchmark and algorithm are available at https://github.com/TuSimple/LiDAR_SOT/. A video demonstration is at https://www.youtube.com/watch?v=BpHixKs91i8.

Model information can be used to predict future trajectories, so it has huge potential to avoid dangerous regions when applying reinforcement learning (RL) on real-world tasks, like autonomous driving. However, existing studies mostly use model-free constrained RL, which causes inevitable constraint violations. This paper proposes a model-based feasibility enhancement technique of constrained RL, which enhances the feasibility of policy using generalized control barrier function (GCBF) defined on the distance to constraint boundary. By using the model information, the policy can be optimized safely without violating actual safety constraints, and the sample efficiency is increased. The infeasibility in solving the constrained policy gradient is handled by an adaptive coefficient mechanism. We evaluate the proposed method in both simulations and real vehicle experiments in a complex autonomous driving collision avoidance task. The proposed method achieves up to four times fewer constraint violations and converges 3.36 times faster than baseline constrained RL approaches.

Currently, most table tennis robots concentrate on the canonical position control problem while ignoring the actual velocity control requirements. In this paper, we consider these requirements and propose a new table tennis robot framework. First, a tailor-made mechanical structure is designed such that the robot can reach large workspaces. Thereafter, in the table tennis trajectory prediction process, a clustering algorithm is introduced to screen the heterogeneous predicted hitting points and filter the invalid ones, thereby significantly improving the prediction accuracy. By using quintic polynomial trajectory planning, smooth and stable high-speed control of the robot hitting motion can be obtained. Finally, a position-based strategy and a velocity-based strategy are devised for returning the table tennis. Extensive experiments demonstrate that the accuracy of the ball's trajectory prediction algorithm is more than 92%. The success rate of returning the ball exceeds 95% at the ball velocity of 3-7 m/s, and the velocity-based strategy performs better compared with the position-based approach at the ball velocity of 7-9 m/s.