IROS 2018 Papers

Long-term visual localization in outdoor environment is a challenging problem, especially faced with the cross-seasonal, bi-directional tasks and changing environment. In this paper we propose a novel visual inertial localization framework that localizes against the LiDAR-built map. Based on the geometry information of the laser map, a hybrid bundle adjustment framework is proposed, which estimates the poses of the cameras with respect to the prior laser map as well as optimizes the state variables of the online visual inertial odometry system simultaneously. For more accurate crossmodal data association, the laser map is optimized using multisession laser and visual data to extract the salient and stable subset for visual localization. To validate the efficiency of the proposed method, we collect data in south part of our campus in different seasons, along the same and opposite-direction route. In all sessions of localization data, our proposed method gives satisfactory results, and shows the superiority of the hybrid bundle adjustment and map optimization1.

Measuring the quality of roads and ensuring they are ready for autonomous driving is important for future transportation systems. Here we focus on developing metrics and algorithms to assess lane marking (LM)qualities from an egocentric view of an inspection vehicle equipped with a global positioning system (GPS)receiver, a frontal-view camera, and a light detection and ranging (LIDAR)system. We propose three quality metrics for LMs: correctness, shape, and visibility. The correctness metric measures the divergence between the expected LMs based on prior map inputs and the actual sensor inputs. The shape metric evaluates smoothness in road curvature and width range. The visibility metric evaluates the contrast between LMs and background road surfaces. We propose a dual-modal algorithm to compute these metrics. We have implemented the algorithms and tested them under KITTI dataset. The results show that our metrics can successfully detect LM anomalies in all testing scenarios.

The growing popularity of autonomous systems creates a need for reliable and efficient metric pose retrieval algorithms. Currently used approaches tend to rely on nearest neighbor search of binary descriptors to perform the 2D-3D matching and guarantee realtime capabilities on mobile platforms. These methods struggle, however, with the growing size of the map, changes in viewpoint or appearance, and visual aliasing present in the environment. The rigidly defined descriptor patterns only capture a limited neighborhood of the keypoint and completely ignore the overall visual context. We propose LandmarkBoost - an approach that, in contrast to the conventional 2D-3D matching methods, casts the search problem as a landmark classification task. We use a boosted classifier to classify landmark observations and directly obtain correspondences as classifier scores. We also introduce a formulation of visual context that is flexible, efficient to compute, and can capture relationships in the entire image plane. The original binary descriptors are augmented with contextual information and informative features are selected by the boosting framework. Through detailed experiments, we evaluate the retrieval quality and performance of Landmark-Boost, demonstrating that it outperforms common state-of-the-art descriptor matching methods.

This paper presents a method that allows a snake robot to climb a ladder. We propose a ladder climbing method for a snake robot that has a smooth surface shape. We design a novel gait for the snake using a gait design method that configures the target form of the snake robot by connecting simple shapes. The climbing motion is executed via shift control and the corresponding motion required to catch the next step on the ladder. In addition, we developed a snake robot that has a smooth exterior body surface through construction of pectinate-shaped parts of the links. We demonstrated the effectiveness of both the proposed gait and the design of the snake robot experimentally.

This paper presents the formalization of the closest point plane representation and an analysis of its incorporation in 3D indoor simultaneous localization and mapping (SLAM). We present a singularity free plane factor leveraging the closest point plane representation, and demonstrate its fusion with inertial preintegratation measurements in a graph-based optimization framework. The resulting LiDAR-inertial 3D plane SLAM (LIPS) system is validated both on a custom made LiDAR simulator and on a real-world experiment.

Higher level functionality in autonomous driving depends strongly on a precise motion estimate of the vehicle. Powerful algorithms have been developed. However, their great majority focuses on either binocular imagery or pure LIDAR measurements. The promising combination of camera and LIDAR for visual localization has mostly been unattended. In this work we fill this gap, by proposing a depth extraction algorithm from LIDAR measurements for camera feature tracks and estimating motion by robustified keyframe based Bundle Adjustment. Semantic labeling is used for outlier rejection and weighting of vegetation landmarks. The capability of this sensor combination is demonstrated on the competitive KITTI dataset, achieving a placement among the top 15. The code is released to the community.

In this paper we present an extension of Direct Sparse Odometry (DSO) [1] to a monocular visual SLAM system with loop closure detection and pose-graph optimization (LDSO). As a direct technique, DSO can utilize any image pixel with sufficient intensity gradient, which makes it robust even in featureless areas. LDSO retains this robustness, while at the same time ensuring repeatability of some of these points by favoring corner features in the tracking frontend. This repeatability allows to reliably detect loop closure candidates with a conventional feature-based bag-of-words (BoW) approach. Loop closure candidates are verified geometrically and Sim(3) relative pose constraints are estimated by jointly minimizing 2D and 3D geometric error terms. These constraints are fused with a co-visibility graph of relative poses extracted from DSO's sliding window optimization. Our evaluation on publicly available datasets demonstrates that the modified point selection strategy retains the tracking accuracy and robustness, and the integrated pose-graph optimization significantly reduces the accumulated rotation-, translation- and scale-drift, resulting in an overall performance comparable to state-of-the-art feature-based systems, even without global bundle adjustment.

AI knowledge representation and reasoning methods consider actions to be blackboxes that abstract away from how they are executed. This abstract view does not suffice for the decision making capabilities required by robotic agents that are to accomplish manipulation tasks. Such robots have to reason about how to pour without spilling, where to grasp a pot, how to open different containers, and so on. To enable such reasoning it is necessary to consider how objects are perceived, how motions can be executed and parameterized, and how motion parameterization affects the physical effects of actions. To this end, we propose to complement and extend symbolic reasoning methods with KnowRobSIM, an additional reasoning infrastructure based on modern game engine technology, including the subsymbolic world modeling through data structures, action simulation based on physics engine, and world scene rendering. We demonstrate how KnowRobSIM can perform powerful reasoning, prediction, and learning tasks that are required for informed decision making in object manipulation.

Technological developments call for increasing perception and action capabilities of robots. Among other skills, vision systems that can adapt to any possible change in the working conditions are needed. Since these conditions are unpredictable, we need benchmarks which allow to assess the generalization and robustness capabilities of our visual recognition algorithms. In this work we focus on robotic kitting in unconstrained scenarios. As a first contribution, we present a new visual dataset for the kitting task. Differently from standard object recognition datasets, we provide images of the same objects acquired under various conditions where camera, illumination and background are changed. This novel dataset allows for testing the robustness of robot visual recognition algorithms to a series of different domain shifts both in isolation and unified. Our second contribution is a novel online adaptation algorithm for deep models, based on batch-normalization layers, which allows to continuously adapt a model to the current working conditions. Differently from standard domain adaptation algorithms, it does not require any image from the target domain at training time. We benchmark the performance of the algorithm on the proposed dataset, showing its capability to fill the gap between the performances of a standard architecture and its counterpart adapted offline to the given target domain.

As personal autonomous mobility is getting to be more widely adopted, it is more important to consider comfortability of stuffs and persons carried by such mobility. In this work, we define the comfort of a trajectory as forces, specifically, translational force, received to objects carried by a robot while following the trajectory by measuring impulse. To maximize such a comfort, we propose a novel, kinodynamic comfort path planning method based on our definition of comfort. Our work is based on direct collocation method for handling our nonconvex objective function. We also introduce Bidirectional Obstacle Detection(BOD)that identifies the distances along the perpendicular directions to the trajectory. This is mainly designed for avoiding obstacles while minimizing forces causing discomfort. Our experimental results show that our method can compute trajectories whose comfort measures can be up to 18 times higher than those computed by prior related objectives, e.g., squared velocity used for generating smooth trajectory.

The transfer of a robot skill between different geometric environments is non-trivial since a wide variety of environments exists, sensor observations as well as robot motions are high-dimensional, and the environment might only be partially observed. We consider the problem of extracting a low-dimensional description of the manipulated environment in form of a kinematic model. This allows us to transfer a skill by defining a policy on a prototype model and morphing the observed environment to this prototype. A deep neural network is used to map depth image observations of the environment to morphing parameter, which include transformations and configurations of the prototype model. Using the concatenation property of affine transformations and the ability to convert point clouds to depth images allows to apply the network in an iterative manner. The network is trained on data generated in a simulator and on augmented data that is created with its own predictions. The algorithm is evaluated on different tasks, where it is shown that iterative predictions lead to a higher accuracy than one-step predictions.

Finding the parameters of a vignetting function for a camera currently involves the acquisition of several images in a given scene under very controlled lighting conditions, a cumbersome and error-prone task where the end result can only be confirmed visually. Many computer vision algorithms assume photoconsistency, constant intensity between scene points in different images, and tend to perform poorly if this assumption is violated. We present a real-time online vignetting and response calibration with additional exposure estimation for global-shutter color cameras. Our method does not require uniformly illuminated surfaces, known texture or specific geometry. The only assumptions are that the camera is moving, the illumination is static and reflections are Lambertian. Our method estimates the camera view poses by sparse visual SLAM and models the vignetting function by a small number of thin plate splines (TPS) together with a sixth-order polynomial to provide a dense estimation of attenuation from sparsely sampled scene points. The camera response function (CRF) is jointly modeled by a TPS and a Gamma curve. We evaluate our approach on synthetic datasets and in real-world scenarios with reference data from a Structure-from-Motion (SfM) system. We show clear visual improvement on textured meshes without the need for extensive meshing algorithms. A useful calibration is obtained from a few keyframes which makes an on-the-fly deployment conceivable.

Optical images provide rich features but require extensive computation resources to process for SLAM. When there are limited computation resources on the robots, it becomes a heavy burden to process the images in real-time. This paper presents the design and implementation of key-frame selection algorithm for multiple robots simultaneous localization and tracking on the multi-robot soccer games which have pre-defined field and objects. Compared to traditional key-frame selection algorithms, this work makes use of the temporal and spatial relationship among objects on the pre-defined field to compute the information entropy. The selection ratio can be adjusted by two parameters: entropy threshold and the maximum moving distance. The experimental results show that the developed method can effectively detect the change of scene using selected key-frames. And comparing with the localization results using all the images, using less than 20% of all images after walking 11,203mm it only increase up to 0.87% trajectory errors.

Many of today's Visual-Inertial Odometry (VIO)frameworks work well under regular motion but have issues and need special treatment under special motion. Here, special does not imply bad or corrupted data but stands for increased difficulty to treat clean data. Common special motion for VIO are large feature displacement due to fast motion close to a scene and zero motion phases not providing sufficient baseline. In this paper we present a feature and frame selection approach which seamlessly handles all motion scenarios without the need of (error prone)motion case identification and subsequent case-specific heuristics. We further show that this approach allows to eliminate features in the state vector (persistent features)altogether while still being able to inherently handle zero motion phases. This reduces computational complexity while maintaining the ability to hover in place. We integrate our frame selection approach into our own VIO algorithm and compare its performance against three state-of-the-art algorithms with real data on a real platform. While our approach shows slightly higher global drift it is the only algorithm that can reliably estimate the pose over a large motion spectrum from fast scale change down to zero motion.

Learning motion primitives from demonstration requires the human demonstrator to effectively relay the task intent to the robot controller. When the task intent is not reflected sufficiently by the demonstration, multiple iterations are required to recover the underlying intent of the demonstrations. However, a large number of iterations can be expensive and might not be practical for each new task. A challenge is that human-in-the-loop demonstrations can be affected by the human motor dynamics (e.g., from visual observation to hand motion), which can lead to differences between the demonstration and intent. The main contribution of this article is to correct for the human motor dynamics and infer the intended action (motion primitive) from the human demonstrations. The proposed approach uses a kernel-based regression approach to learn the inverse human-dynamics response. These models are then used to correct for human-motor-dynamics and infer the intent of the human-in-the-loop demonstrator. Experimental validation is performed with an assisted teleoperation setup where the underlying intent is specified using an augmented reality display. Results indicate that the proposed approach leads to more precise intent estimation as compared to the actual human demonstrations.

External force observer for humanoid robots has been widely studied in the literature. However, most of the proposed approaches generally rely on information from six-axis force/torque sensors, which the small or medium-sized humanoid robots usually do not have. As a result, those approaches cannot be applied to this category of humanoid robots, which is widely used nowadays in education or research. In this paper, we improve the external force observer in [1] to handle the case of an external force applied in any direction and at an arbitrary point of the robot structure. The new observer is based on Kalman filter formulation and it allows the estimation of the three force components. The observer is simple to implement and can easily run in real time using the embedded processor of a medium-sized humanoid robot such as Nao or Darwin-OP. Moreover, the observer does not require any change to the robot hardware as it only uses measurements from the available force-sensing resistors (FSR) inserted under the feet of the humanoid robot and from the robot inertial measurement unit (IMU). The proposed observer was extensively validated on a Nao humanoid robot. In all conducted experiments, the observer successfully estimated the external force within a reasonable margin of error.

Emergency trajectories enable one to move faster through an environment while still moving safely. Having an emergency trajectory within an observed vacant space makes it possible to safely navigate through unknown territory or through a door without slowing down. Emergency trajectories allow for safe navigation of a vehicle into a safe system state, e.g. a stop, in the event of recognition of an obstacle. This work formally proves the benefit of using emergency trajectories to generate safe and faster motion controls as compared to vehicle operation without such trajectories. Furthermore, this work also presents a working integration of this formalism into a vehicle's low level control system in a Moving Horizon Trajectory Planner (MHTP) with an update rate of 10Hz. Using an MHTP along with a dynamic model of the environment and the proposed constraints, the system is able to derive emergency trajectory candidates which fulfill our safety requirements. This distinguishes the approach from that of others, which replans discrete paths that are then followed by the vehicle's local control system. This approach was implemented on a differential-drive mobile agent and tested using non-static environment assumptions. Simulated and real-robot experimental results illustrate the quality of our approach.

To improve the movement ability of humanoid robots, instead of traditional methods dependent on only power of actuators, there is possibility that utilizing elasticity inspired from collaboration of muscle and tendon of human is effective to achieve high-power movement. In this study, we aimed to realize a jumping motion that accumulates energy more appropriately in spring by combining active joint driving with spring behavior like human tendons and muscles. We proposed a countermovement jump method using the resonance with the leg's active pushing-off movement and leg stiffness. To achieve active pushing-off and joint stiffness, we developed a new joint mechanism using leaf springs and an actuator unit with a worm gear. We then performed experiments to evaluate the effectiveness of the proposed mechanism and methods. Finally, the robot achieved a countermovement jump using active kicking and leg's elasticity.

Applying agrochemicals is the default procedure for conventional weed control in crop production, but has negative impacts on the environment. Robots have the potential to treat every plant in the field individually and thus can reduce the required use of such chemicals. To achieve that, robots need the ability to identify crops and weeds in the field and must additionally select effective treatments. While certain types of weed can be treated mechanically, other types need to be treated by (selective) spraying. In this paper, we present an approach that provides the necessary information for effective plant-specific treatment. It outputs the stem location for weeds, which allows for mechanical treatments, and the covered area of the weed for selective spraying. Our approach uses an end-to-end trainable fully convolutional network that simultaneously estimates stem positions as well as the covered area of crops and weeds. It jointly learns the class-wise stem detection and the pixel-wise semantic segmentation. Experimental evaluations on different real-world datasets show that our approach is able to reliably solve this problem. Compared to state-of-the-art approaches, our approach not only substantially improves the stem detection accuracy, i.e., distinguishing crop and weed stems, but also provides an improvement in the semantic segmentation performance.