IROS 2023 Papers

Pouring snacks and moving containers with beverages are challenging for a service robot. To obtain accurate content properties for planning robotic motion, tactile sensing can provide information about the pressure distribution of the contact surface, which is not obvious by visual observation. In this work, we focus on estimating the content properties of various content materials in distinct deformable containers through tactile interactions. We propose a learning-based model that can estimate content properties by using the tactile data collected by slightly squeezing a container with the content of interest. We analyzed an uncalibrated tactile sensor and collected a dataset consisting of 1125 sets of tactile sequences, which are combinations of five types of deformable containers and eleven types of content materials in different content heights. Experiments were conducted on content estimation with known contents and containers, unknown contents, and unknown containers. For unknown contents, our model can still achieve 8.5% height relative error and 79.7% state of matter accuracy. Furthermore, we analyzed that the tactile features of contents with similar content properties are close in the latent snace to show the effectiveness of our model.

Deploying humans in a high-risk environment to extract casualties in order to provide medical attention is an inherently dangerous endeavor. To minimize this risk, Robotics and Autonomous Systems can be deployed in hazardous areas in place of human personnel to limit the exposure of first responders to various life-threatening conditions. The success of robotic extraction of injured persons depends heavily on how safely the human subject is handled. Therefore, the integration of intelligent technologies for secure control and motion planning is crucial in overcoming the dynamic and complex challenges of robotic grasping and manipulation. In this regard, the measurement of the target human subject's weight is an essential factor for safe grasping and maneuvering during robotic interactions with humans. This paper presents a contactless vision-based approach for estimating the weight of the human body. This approach employs visual body perception, 3D body point cloud representation, and a deep learning network for body segmentation to measure specific body parameters. Next, the body parameters are fed into a neural network model to predict the total body weight. This prediction then enables an approximation of the weight of individual body segments to be obtained.

Studying the manipulation of deformable linear objects has significant practical applications in industry, including car manufacturing, textile production, and electronics automation. However, deformable linear object manipulation poses a significant challenge in developing planning and control algorithms, due to the precise and continuous control required to effectively manipulate the deformable nature of these objects. In this paper, we propose a new framework to control and maintain the shape of deformable linear objects with two robot manipulators utilizing environmental contacts. The framework is composed of a shape planning algorithm which automatically generates appropriate positions to place fixtures, and an object-centered skill engine which includes task and motion planning to control the motion and force of both robots based on the object status. The status of the deformable linear object is estimated online utilizing visual as well as force information. The framework manages to handle a cable routing task in real-world experiments with two Panda robots and especially achieves contact-aware and flexible clip fixing with challenging fixtures.

Non-prehensile manipulation methods usually use a simple end effector, e.g., a single rod, to manipulate the object. Compared to the grasping method, such an end effector is compact and flexible, and hence it can perform tasks in a constrained workspace; As a trade-off, it has relatively few degrees of freedom (DoFs), resulting in an under-actuation problem with complex constraints for planning and control. This paper proposes a new non-prehensile manipulation method for the task of object retrieval in cluttered environments, using a rod-like pusher. Specifically, a candidate trajectory in a cluttered environment is first generated with an improved Rapidly-Exploring Random Tree (RRT) planner; Then, a Model Predictive Control (MPC) scheme is applied to stabilize the slider's poses through necessary contact with obstacles. Different from existing methods, the proposed approach is with the contact-aware feature, which enables the synthesized effect of active removal of obstacles, avoidance behavior, and switching contact face for improved dexterity. Hence both the feasibility and efficiency of the task are greatly promoted. The performance of the proposed method is validated in a planar object retrieval task, where the target object, surrounded by many fixed or movable obstacles, is manipulated and isolated. Both simulation and experimental results are presented.

In sim-to-real Reinforcement Learning (RL), a policy is trained in a simulated environment and then deployed on the physical system. The main challenge of sim-to-real RL is to overcome the reality gap - the discrepancies between the real world and its simulated counterpart. Using generic geometric representations, such as convex decomposition, triangular mesh, signed distance field can improve simulation fidelity, and thus potentially narrow the reality gap. Common to these approaches is that many contact points are generated for geometrically-complex objects, which slows down simulation and may cause numerical instability. Contact reduction methods address these issues by limiting the number of contact points, but the validity of these methods for sim-to-real RL has not been confirmed. In this paper, we present a contact reduction method with bounded stiffness to improve the simulation accuracy. Our experiments show that the proposed method critically enables training RL policy for a tight-clearance double pin insertion task and successfully deploying the policy on a rigid, position-controlled physical robot.

Object-based maps are relevant for scene under-standing since they integrate geometric and semantic information of the environment, allowing autonomous robots to robustly localize and interact with on objects. In this paper, we address the task of constructing a metric-semantic map for the purpose of long-term object-based localization. We exploit 3D object detections from monocular RGB frames for both, the object-based map construction, and for globally localizing in the constructed map. To tailor the approach to a target environment, we propose an efficient way of generating 3D annotations to finetune the 3D object detection model. We evaluate our map construction in an office building, and test our long-term localization approach on challenging sequences recorded in the same environment over nine months. The experiments suggest that our approach is suitable for constructing metric-semantic maps, and that our localization approach is robust to long-term changes. Both, the mapping algorithm and the localization pipeline can run online on an onboard computer. We release an open-source C++/ros implementation of our approach.

Effective design automation for building robots would make development faster and easier while also less prone to design errors. However, complex multi-domain constraints make creating such tools difficult. One persistent challenge in achieving this goal of design automation is the fundamental problem of component selection, an optimization problem where, given a general robot model, components must be selected from a possibly large set of catalogs to minimize design objectives while meeting target specifications. Different approaches to this problem have used Monotone Co-Design Problems (MCDPs) or linear and quadratic programming, but these require judicious system approximations that affect the accuracy of the solution. We take an alternative approach formulating the component selection problem as a combinatorial optimization problem, which does not require any system approximations, and using constraint programming (CP) to solve this problem with a depth-first branch-and-bound algorithm. As the efficacy of CP critically depends upon the orderings of variables and their domain values, we present two heuristics specific to the problem of component selection that significantly improve solve time compared to traditional constraint satisfaction programming heuristics. We also add redundant constraints to the optimization problem to further improve run time by evaluating certain global constraints before all relevant variables are assigned. We demonstrate that our CP approach can find optimal solutions from over 20 trillion candidate solutions in only seconds, up to 48 times faster than an MCDP approach solving the same problem. Finally, for three different robot designs we build the corresponding robots to physically validate that the selected components meet the target design specifications.

The field of robotic Flexible Endoscopes (FEs) has progressed significantly, offering a promising solution to reduce patient discomfort. However, the limited autonomy of most robotic FEs results in non-intuitive and challenging manoeuvres, constraining their application in clinical settings. While previous studies have employed lumen tracking for autonomous navigation, they fail to adapt to the presence of obstructions and sharp turns when the endoscope faces the colon wall. In this work, we propose a Deep Reinforcement Learning (DRL)-based navigation strategy that eliminates the need for lumen tracking. However, the use of DRL methods poses safety risks as they do not account for potential hazards associated with the actions taken. To ensure safety, we exploit a Constrained Reinforcement Learning (CRL) method to restrict the policy in a predefined safety regime. Moreover, we present a model selection strategy that utilises Formal Verification (FV) to choose a policy that is entirely safe before deployment. We validate our approach in a virtual colonoscopy environment and report that out of the 300 trained policies, we could identify three policies that are entirely safe. Our work demonstrates that CRL, combined with model selection through FV, can improve the robustness and safety of robotic behaviour in surgical applications.

Most state-of-the-art data-driven grasp sampling methods propose stable and collision-free grasps uniformly on the target object. For bin-picking, executing any of those reachable grasps is sufficient. However, for completing specific tasks, such as squeezing out liquid from a bottle, we want the grasp to be on a specific part of the object's body while avoiding other locations, such as the cap. This work presents a generative grasp sampling network, VCGS, capable of constrained 6-Degrees of Freedom (DoF) grasp sampling. In addition, we also curate a new dataset designed to train and evaluate methods for constrained grasping. The new dataset, called CONG, consists of over 14 million training samples of synthetically rendered point clouds and grasps at random target areas on 2889 objects. VCGS is benchmarked against GraspNet, a state-of-the-art unconstrained grasp sampler, in simulation and on a real robot. The results demonstrate that VCGS achieves a 10-15% higher grasp success rate than the baseline while being 2–3 times as sample efficient. Supplementary material is available on our project website.

Dynamic movement primitives are widely used for learning skills that can be demonstrated to a robot by a skilled human or controller. While their generalization capabilities and simple formulation make them very appealing to use, they possess no strong guarantees to satisfy operational safety constraints for a task. We present constrained dynamic movement primitives (CDMPs), which can allow for positional constraint satisfaction in the robot workspace. Our method solves a non-linear optimization to perturb an existing DMP's forcing weights to admit a Zeroing Barrier Function (ZBF), which certifies positional workspace constraint satisfaction. We demonstrate our approach under different positional constraints on the end-effector movement on multiple physical robots, such as obstacle avoidance and workspace limitations.

Inertia drift is an aggressive transitional driving maneuver, which is challenging due to the high nonlinearity of the system and the stringent requirement on control and planning performance. This paper presents a solution for the consecutive inertia drift of an autonomous RC car based on primitive-based planning and data-driven control. The planner generates complex paths via the concatenation of path segments called primitives, and the controller eases the burden on feedback by interpolating between multiple real trajectories with different initial conditions into one near-feasible reference trajectory. The proposed strategy is capable of drifting through various paths containing consecutive turns, which is validated in both simulation and reality.

Object rearrangement is the problem of enabling a robot to identify the correct object placement in a complex environment. Prior work on object rearrangement has explored a diverse set of techniques for following user instructions to achieve some desired goal state. Logical predicates, images of the goal scene, and natural language descriptions have all been used to instruct a robot in how to arrange objects. In this work, we argue that burdening the user with specifying goal scenes is not necessary in partially-arranged environments, such as common household settings. Instead, we show that contextual cues from partially arranged scenes (i.e., the placement of some number of pre-arranged objects in the environment) provide sufficient context to enable robots to perform object rearrangement without any explicit user goal specification. We introduce ConSOR, a Context-aware Semantic Object Rearrangement framework that utilizes contextual cues from a partially arranged initial state of the environment to complete the arrangement of new objects, without explicit goal specification from the user. We demonstrate that ConSOR strongly outperforms two baselines in generalizing to novel object arrangements and unseen object categories. The code and data are available at https://github.com/kartikvrama/consor.

We investigate motion planning algorithms for the assembly of shapes in the tilt model in which unit-square tiles move in a grid world under the influence of uniform external forces and self-assemble according to certain rules. We provide several heuristics and experimental evaluation of their success rate, solution length, and runtime.

Designing robots is a multiphase process aimed at solving a multi-criteria optimization problem to find the best possible detailed design. Generative design (GD) aims to accelerate the design process compared to manual design, since GD allows exploring and exploiting the vast design space more efficiently. In the field of robotics, however, relevant research focuses mostly on the generation of fully-actuated open chain kinematics, which is trivial in mechanical engineering perspective. Within this paper, we address the problem of generative design of closed-chain linkage mechanisms. A GD algorithm has to be able to generate meaningful mechanisms which satisfy conditions of existence. We propose an optimization-driven algorithm for generation of planar closed-chain linkages to follow a predefined trajectory. The algorithm creates an unlimited range of physically reproducible design alternatives that can be further tested in simulation. These tests could be done in order to find solutions that satisfy extra criteria, e.g., desired dynamic behavior or low energy consumption. The proposed algorithm is called “respawn” since it builds a new linkage after the ancestor has been tested in a virtual environment in pursuit for the optimal solution. To show that the algorithm is general enough, we show a set of generated linkages that can be used for a wide class of robots.

Autonomous mobile robots (AMRs) face challenges in efficiently covering complex environments. To navigate narrow and expansive areas, AMRs must have two essential attributes: compact size for confined spaces and larger size with omnidirectional locomotion for broader spaces. This study utilizes omnidirectional expand and collapse robots (OECRs) to demonstrate efficient area coverage. OECRs can collapse to navigate through confined spaces and expand for efficient coverage in broad spaces. However, current complete coverage path planning (CCPP) methods do not account for the expanded and collapsed states of OECRs. To address this, a depth-first search (DFS) approach is proposed for OECRs' CCPP, which can adjust the robotic footprint along the CCPP path to reduce path length. The proposed DFS outperforms the state-of-the-art CCPP in terms of increased area coverage and reduced distance traveled on a selected map.

This paper studies the problem of Competitive Ant Coverage, in which two ant-like robots with very limited capabilities in terms of sensing range, computational power, and knowledge of the world compete in an area coverage task. We examine two variants of the problem that differ in the robot's objective: either being the First to Cover a Cell (FCC), or being the Last to Cover a Cell (LCC). Each robot's goal is to acquire (by visiting first or last, respectively) more cells than the opposing robot, and by that win the game. We examine the problem both theoretically and empirically, and show that the main strategy for dominance revolves around the ability to pursue: in LCC, we wish to pursue the opposing robot, whereas in FCC, we wish to create a scenario wherein the opposing robot pursues us. We find that this ability relies more heavily on knowledge of the opponent's strategy than on the robot's sensing capabilities. Moreover, given the robot's limited capabilities, we find that this knowledge-gap cannot be easily mitigated by learning.

Manual materials handling occupations often involve repetitive lifting, lowering, and carrying motions, which can lead to muscular fatigue and/or injury. The risk increases when loads must be worn on the body for the entirety of a job shift. Exoskeletons have been developed to assist these types of motions, but require the user to bear the weight of a load through their body. Load carriage exoskeletons have been developed to offload worn mass from the user to the ground through the device structure, but they have had limited success and have not been well studied in manual materials handling tasks. In this paper, we introduce a hip-knee-ankle exoskeleton and two control methods: virtual model control and gravity compensation. We compared the ability of each controller to reduce lower-limb muscle activity during squatting. Because the virtual model controller is tailored to squatting, we hypothesized that it would outperform gravity compensation. Both controllers were able to reduce the activity of major lower-limb muscle groups during squatting when compared to squatting with the exoskeleton turned off. Contrary to our original hypothesis, the gravity compensation controller generally outperformed the virtual model controller, which may have been caused by the gravity compensation controller having more consistent knee torque application and the virtual model controller requiring better per-user tuning and familiarization. These results indicate the efficacy of both controllers in reducing injury risk in the lower limbs during squatting.

To alleviate the sample complexity of reinforcement learning algorithms, simulations are a common approach to train control policies before deploying the policy on a real-world robot. However, a gap between simulation and reality generally persists, which endorses the aim to train robust policies already in simulation such that those can be transferred to a real robot at a high success rate. In this paper, we investigate history-dependent policies for drone control in the context of zero-shot transfer learning, where the training is conducted exclusively in simulation. We compare policies represented by feed-forward neural networks with recurrent neural networks and assess both performance and robustness on a real-world quadrotor. Furthermore, we study if an end-to-end learned representation can control a quadrotor based on raw onboard-sensor information only, rendering accurate state estimation from a Kalman filter obsolete. Our results show that recurrent control policies achieve similar performance and robustness as feed-forward policies when acting on state estimates. With raw sensory data, however, recurrent networks offer higher success rates for sim-to-real transfer than feed-forward networks. We also find that recurrent architectures are advantageous when system parameters such as latency are uncertain.

The goal of this work is to develop a fast, compact, accurate, and robust neural network for automatic thyroid gland segmentation in medical ultrasound images when the computational resources are limited, such as portable intelligent medical diagnostic devices or robots. An adaptive width radial basis function neural network model with discriminative features is proposed in this work. The model combines the local characteristics and global context of the entire feature space, which improves the performance of the automatic thyroid gland segmentation, and enhances the data fitting capability and robustness of the neural network. A new feature selection method combining the measurements of area under the curve values and Pearson correlation coefficients is proposed to select six discriminative and low correlated features from a set of handcrafted features, which helps to construct a compact yet accurate radial basis function neural network. The proposed method is experimented on thyroid ultrasound images using five-fold cross validation. The proposed method achieves an average accuracy of 0.9770 and an average IoU of 0.7841, achieving competitive segmentation performance compared to convolutional neural networks, with thousands of times faster training speed on a CPU.