IROS 2021 Papers

Human-robot collaboration is an essential re-search topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g., “go there” with a pointing gesture. To bridge the gap and fill in the missing dimension of communication in embodied agent navigation, we propose investigating the effects of using gestures as the communicative interface instead of verbal cues. Specifically, we develop a VR-based 3D simulation environment, named Gesture-based THOR (Ges-THOR), based on AI2-THOR platform. In this virtual environment, a human player is placed in the same virtual scene and shepherds the artificial agent using only gestures. The agent is tasked to solve the navigation problem guided by natural gestures with unknown semantics; we do not use any predefined gestures due to the diversity and versatile nature of human gestures. We argue that learning the semantics of natural gestures is mutually beneficial to learning the navigation task—learn to communicate and communicate to learn. In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.

Collaborative robots are expected to work alongside humans and directly replace human workers in some cases, thus effectively responding to rapid changes in assembly lines. Current methods for programming contact-rich tasks, particularly in heavily constrained spaces, tend to be fairly inefficient. Therefore, faster and more intuitive approaches are urgently required for robot teaching. This study focuses on combining visual servoing-based learning from demonstration (LfD) and force-based learning by exploration (LbE) to enable the fast and intuitive programming of contact-rich tasks with minimal user efforts. Two learning approaches were developed and integrated into a framework, one relying on human-to-robot motion mapping (visual servoing approach) and the other relying on force-based reinforcement learning. The developed framework implements the noncontact demonstration teaching method based on the visual servoing approach and optimizes the demonstrated robot target positions according to the detected contact state. The developed framework is compared with two most commonly used baseline techniques, i.e., teach pendant-based teaching and hand-guiding teaching. Furthermore, the efficiency and reliability of the framework are validated via comparison experiments involving the teaching and execution of contact-rich tasks. The proposed framework shows the best performance in terms of the teaching time, execution success rate, risk of damage, and ease of use.

This work proposes a model predictive control approach of a wheeled mobile robot based on a local-minima free navigation function. The constructed navigation function includes information on a goal location and obstacles. Novel conservative navigation is introduced that is simple to compute and yields convergent control behavior. To solve the optimization problem the combined optimization is proposed by a fixed candidate set and particle swarm optimization. The efficiency of the proposed approaches is validated on simulations, by experimental results on the Husky A200 mobile robot and comparisons to Stable Sparse RRT planner.

This paper investigates servicing waypoints in a wide area using collaborative deployments of vehicles with heterogeneous range and mobility constraints. We formulate a joint planning problem for a single transport truck and multiple service drones in which the truck is constrained to a road and must deploy a team of range-constrained drones to visit waypoints. The need to deploy, collect, and redeploy drones over multiple flights introduces both route finding and scheduling aspects to this problem. We solve large problem instances by decoupling our approach into a service drone route finding phase and a transport truck scheduling phase. Numerical simulations explore the qualitative character of the driving schedule and the quantitative marginal value of adding additional drones to the team as a function of agent number and relative speed. The combination of road network constraints and range constraints make this problem especially relevant to wide area forestry, last-mile delivery, and ecological monitoring applications.

Human-robot collaboration in shared workspaces enables companies to improve efficiency and the quality of work for human workers. A novel research direction in this field is that human and robot dynamically negotiate which actions to perform. This requires the robot to permanently monitor the current task state and actions the human has performed. We envision a system that tracks the task progress based on visual input. We assume that a task specification is provided in terms of a coloured Petri net. The contribution of this paper is twofold. We introduce the notion of emissions for coloured Petri nets that incorporate partial observability and uncertainty. We then show how one can efficiently determine the evolution of the net when a sequence of partial observations is provided. To this end, we determine a small set of transition as candidates in a firing sequence first. Then, transition sequences are sampled to obtain markings compatible with the latest observation. We evaluate our algorithm in a simulated environment on a pick and place task and compare it to an aging-based approach from literature. Results show that our algorithm achieves higher precision compared to the aging-based approach. Running times indicate that the update procedure can run at 15 Hz on average.

Storytelling plays a central role in human socializing and entertainment, and research on conducting storytelling with robots is gaining interest. However, much of this research assumes that story content is curated. In this paper, we expand the recently-proposed task of collaborative storytelling, where an intelligent agent and a person collaborate to create a unique story by taking turns adding to it, for application to social robot and consider the design implications that arise. Since latency can be detrimental to human-robot interaction, we examine the performance-latency trade-offs of an existing generate-and-rank-based approach to collaborative storytelling by finding the optimal ranker’s sample size that strikes the best balance between quality and computational cost. We improve on existing evaluation that was previously based on system-generated stories by having human participants play the collaborative storytelling game with our system and comparing the stories they create with our system to a naive baseline. Finally, we conduct a pilot elicitation survey that sheds light on issues to consider when adapting our collaborative storytelling system to a social robot. Our evaluation shows that participants have a positive view of collaborative storytelling with a social robot and consider rich, emoting capabilities to be key to enjoyment.

The recent years have seen the increasing importance of cognitive models for improved robot navigation. In this paper, a novel cognitive navigation package, which consists of topometric map representation and a three-level path planner, is proposed. The topometric maps are built from architectural floor plans with additional features within a limited number of selected regions. The inherent discrepancies between floor plans and the robot’s actual sensory readings are handled by the three-level path planner. The unique feature of this approach is that accurate localization is only required at the selected regions. At the other regions the robot will rely on the guiding directions towards the goal rather than on its accurate position on the map for navigation. Experiments show that our approach can endow a robot with capability for semantic interpretation and localization in unseen and dynamic environments.

Robust and accurate localization is an essential component for robotic navigation and autonomous driving. The use of cameras for localization with high definition map (HD Map) provides an affordable localization sensor set. Existing methods suffer from pose estimation failure due to error prone data association or initialization with accurate initial pose requirement. In this paper, we propose a cost-effective vehicle localization system with HD map for autonomous driving that uses cameras as primary sensors. To this end, we formulate vision-based localization as a data association problem that maps visual semantics to landmarks in HD map. Specifically, system initialization is finished in a coarse to fine manner by combining coarse GPS (Global Positioning System) measurement and fine pose searching. In tracking stage, vehicle pose is refined by implicitly aligning the semantic segmentation result between image and landmarks in HD maps with photometric consistency. Finally, vehicle pose is computed by pose graph optimization in a sliding window fashion. We evaluate our method on two datasets and demonstrate that the proposed approach yields promising localization results in different driving scenarios. Additionally, our approach is suitable for both monocular camera and multi-cameras that provides flexibility and improves robustness for the localization system.

In this paper, we study the problem of zero-shot sim-to-real when the task requires both highly precise control with sub-millimetre error tolerance, and wide task space generalisation. Our framework involves a coarse-to-fine controller, where trajectories begin with classical motion planning using ICP-based pose estimation, and transition to a learned end-to-end controller which maps images to actions and is trained in simulation with domain randomisation. In this way, we achieve precise control whilst also generalising the controller across wide task spaces, and keeping the robustness of vision-based, end-to-end control. Real-world experiments on a range of different tasks show that, by exploiting the best of both worlds, our framework significantly outperforms purely motion planning methods, and purely learning-based methods. Furthermore, we answer a range of questions on best practices for precise sim-to-real transfer, such as how different image sensor modalities and image feature representations perform.

We consider the problem of co-designing embodied intelligence as a whole in a structured way, from hardware components such as propulsion systems and sensors to software modules such as control and perception pipelines. We propose a principled approach to formulate and solve complex embodied intelligence co-design problems, leveraging a monotone co-design theory. The methods we propose are intuitive and integrate heterogeneous engineering disciplines, allowing analytical and simulation-based modeling techniques and enabling interdisciplinarity. We illustrate through a case study how, given a set of desired behaviors, our framework is able to compute Pareto efficient solutions for the entire hardware and software stack of a self-driving vehicle.

Motion planning for complex dynamic systems such as helicopters is a challenging problem due to non-holonomic and nonlinear differential constraints. Approaches for optimal kinodynamics motion planning have only been demonstrated for simple dynamic systems such as Dubins car or linear systems. In this paper we present Closed-loop FMT* (CL-FMT*) which is an extension of FMT* [7] that uses closed-loop prediction. With closed-loop prediction the computationally intense two-point-boundary-value steering procedure can be avoided. We describe the effectiveness of CL-FMT* compared to CL-RRT*. We then demonstrate the use of CL-FMT* for helicopter landing trajectory planning with realistic dynamics constrains.

The sense of taste is fundamental to a human chef’s ability to cook tasty food. To develop robots that can demonstrate human-like cooking, robots need to be equipped with a sense of taste and enabled to use this perception capability to improve or understand the food which they are cooking. We propose a first study of using a salinity sensor to provide a robot with a sense of saltiness. We then demonstrate how this artificial taste receptor can be used to create an autonomous closed-loop cooking system that uses a measurement of saltiness to improve the cooking process of preparing scrambled eggs. Specifically, we show that the sensor measurements can be mapped to different taste metrics such as the overall saltiness and state of mixing the eggs, and how the cooking process can be adapted to match a human-cooked dish, hence individual preferences.

Soft climbing robots have been attracting increasing attention in soft robotics community, and a lot of prototypes been proposed with basic climbing function implemented. Climbing on poles is a challenge with soft robots, and the capability of current pole-climbing soft robots needs to be improved in terms of adaptability to various poles and deformation controllability or constraining of the soft body. In this paper, a rigid-compliant coupling or constraint method is proposed in design of pole climbing robots. Specifically, a novel gripper with inner expanding bubbles and a rigid-compliant belt is presented and designed, featured with excellent shape/size-adaptability and grip-reliability. To constrain undesired deformation of the soft body, rigid-compliant belts are also adopted. Three rigid-compliant constrained actuators are connected to implement an Ω-shaped deformation of the body for climbing motion. The kinematic feature of the body is established, and the two main features of the inner expanding gripper are analyzed. A series of experiments, where the robot climbs on poles with different shapes and sizes and in different poses, have verified the effectiveness of the presented rigid-compliant constraint and the shape/size adaptability and grip-reliability of the novel inner expanding grippers.

Replanning is an essential problem for robots operating in a dynamic and complex environment for responsive and robust autonomy. Previous incremental-search algorithms efficiently reuse existing search results to facilitate a new plan when the environment changes. Yet, they rely solely on geometric information of the environment encoded in an edge-weighted graph. However, semantic information often provides valuable insights that cannot easily be captured quantitatively. We encode both semantic and geometric information of the environment in a weighted colored graph, in which the edges are partitioned into a finite set of ordered semantic classes (e.g., colors), and then we incrementally search for the shortest path among the set of paths with minimal inclusion of inferior classes, using information from the previous search using ideas similar to LPA*. The proposed Class-Ordered LPA* (COLPA*) algorithm inherits the strong theoretical properties of LPA*, namely, optimality and efficiency, but optimality now is with respect to the total path order. Numerical examples show that semantic information helps reduce the relevant search space in a dynamic environment.

This paper studies the problem of interaction control based on force and velocity measurements. Without restricting ourselves to fixed control structures such as admittance or impedance control, we consider the general topology of the problem. Adopting the idea of Youla-based 2DOF control, we reveal a generic control architecture that splits the problem into two independent parts: nominal admittance shaping and disturbance rejection. This pinpoints conceptual flaws in classical interaction control architectures and suggests a way to circumvent them. As a by-product, a complete and compact parameterization of all achievable admittances is derived. The potential of the proposed architecture for analysis and controller design is demonstrated and validated experimentally.

We present a challenging dataset, ChangeSim, aimed at online scene change detection (SCD) and more. The data is collected in photo-realistic simulation environments with the presence of environmental non-targeted variations, such as air turbidity and light condition changes, as well as targeted object changes in industrial indoor environments. By collecting data in simulations, multi-modal sensor data and precise ground truth labels are obtainable such as the RGB image, depth image, semantic segmentation, change segmentation, camera poses, and 3D reconstructions. While the previous online SCD datasets evaluate models given well-aligned image pairs, ChangeSim also provides raw unpaired sequences that present an opportunity to develop an online SCD model in an end-to-end manner, considering both pairing and detection. Experiments show that even the latest pair-based SCD models suffer from the bottleneck of the pairing process, and it gets worse when the environment contains the non-targeted variations. Our dataset is available at https://sammica.github.io/ChangeSim/.

We study the problem of multi-robot mapless navigation in the popular Centralized Training and Decentralized Execution (CTDE) paradigm. This problem is challenging when each robot considers its path without explicitly sharing observations with other robots and can lead to non-stationary issues in Deep Reinforcement Learning (DRL). The typical CTDE algorithm factorizes the joint action-value function into individual ones, to favor cooperation and achieve decentralized execution. Such factorization involves constraints (e.g., monotonicity) that limit the emergence of novel behaviors in an individual as each agent is trained starting from a joint action-value. In contrast, we propose a novel architecture for CTDE that uses a centralized state-value network to compute a joint state-value, which is used to inject global state information in the value-based updates of the agents. Consequently, each model computes its gradient update for the weights, considering the overall state of the environment. Our idea follows the insights of Dueling Networks as a separate estimation of the joint state-value has both the advantage of improving sample efficiency, while providing each robot information whether the global state is (or is not) valuable. Experiments in a robotic navigation task with 2 4, and 8 robots, confirm the superior performance of our approach over prior CTDE methods (e.g., VDN, QMIX).

Category-level 6D pose estimation, aiming to predict the location and orientation of unseen object instances, is fundamental to many scenarios such as robotic manipulation and augmented reality, yet still remains unsolved. Precisely recovering instance 3D model in the canonical space and accurately matching it with the observation is an essential point when estimating 6D pose for unseen objects. In this paper, we achieve accurate category-level 6D pose estimation via cascaded relation and recurrent reconstruction networks. Specifically, a novel cascaded relation network is dedicated for advanced representation learning to explore the complex and informative relations among instance RGB image, instance point cloud and category shape prior. Furthermore, we design a recurrent reconstruction network for iterative residual refinement to progressively improve the reconstruction and correspondence estimations from coarse to fine. Finally, the instance 6D pose is obtained leveraging the estimated dense correspondences between the instance point cloud and the reconstructed 3D model in the canonical space. We have conducted extensive experiments on two well-acknowledged benchmarks of category-level 6D pose estimation, with significant performance improvement over existing approaches. On the representatively strict evaluation metrics of 3D75 and 5°2cm, our method exceeds the latest state-of-the-art SPD [1] by 4.9% and 17.7% on the CAMERA25 dataset, and by 2.7% and 8.5% on the REAL275 dataset. Codes are avaliable at https://wangjiaze.cn/projects/6DPoseEstimation.html.

Casting manipulation has been studied to expand the robot’s movable range. In this manipulation, the robot throws and reaches the end effector to a distant target. Usually, a special casting manipulator, which consists of rigid arm links and specific flexible linear objects, is constructed for an effective casting manipulation. However, the special manipulator cannot perform normal manipulations, such as picking and placing, grasping, and operating objects. We propose that the normal robot arm, which can perform normal tasks, picks up an unknown string in the surrounding environment and realizes casting manipulation with it. As the properties of the string are not provided in advance, it is crucial how to reflect it in casting manipulation. This is realized by the motion generation of the robot arm with the simulation of string movement, actual string manipulation by the robot arm, and string parameter estimation from the actual string movement. After repeating these three steps, the simulated string movement approximates the actual to realize casting manipulation with the unknown string. We confirmed the effectiveness of the proposed method through experiments. The try of this study will lead to enhancement of the performance of home service robot, exploration robot, rescue robot and entertainment robot.