ICRA 2022 论文列表

We achieved contact-rich flexible object manipulation, which was difficult to control with vision alone. In the unzipping task we chose as a validation task, the gripper grasps the puller, which hides the bag state such as the direction and amount of deformation behind it, making it difficult to obtain information to perform the task by vision alone. Additionally, the flexible fabric bag state constantly changes during operation, so the robot needs to dynamically respond to the change. However, the appropriate robot behavior for all bag states is difficult to prepare in advance. To solve this problem, we developed a model that can perform contact-rich flexible object manipulation by real-time prediction of vision with tactility. We introduced a point-based attention mechanism for extracting image features, softmax transformation for predicting motions, and convolutional neural network for extracting tactile features. The results of experiments using a real robot arm revealed that our method can realize motions responding to the deformation of the bag while reducing the load on the zipper. Furthermore, using tactility improved the success rate from 56.7% to 93.3% compared with vision alone, demonstrating the effectiveness and high performance of our method.

We present a novel method for the direct transfer of grasps and manipulations between objects and hands through utilization of contact areas. Our method fully preserves contact shapes, and in contrast to existing techniques, is not dependent on grasp families, requires no model training or grasp sampling, makes no assumptions about manipulator morphology or kinematics, and allows user control over both transfer parameters and solution optimization. Despite these accommodations, we show that our method is capable of synthesizing kinematically-feasible whole hand poses in seconds even for poor initializations or hard-to-reach contacts. We additionally highlight the method's benefits in both response to design alterations as well as fast approximation over in-hand manipulation sequences. Finally, we demonstrate a solution generated by our method on a physical, custom-designed prosthetic hand.

This paper presents Contact Mode Guided Manipulation Planning (CMGMP) for 3D quasistatic and quasi-dynamic rigid body motion planning in dexterous manipulation. The CMGMP algorithm generates hybrid motion plans including both continuous state transitions and discrete contact mode switches, without the need for pre-specified contact sequences or pre-designed motion primitives. The key idea is to use automatically enumerated contact modes of environment-object contacts to guide the tree expansions during the search. Contact modes automatically synthesize manipulation primitives, while the sampling-based planning framework sequences those primitives into a coherent plan. We test our algorithm on fourteen 3D manipulation tasks, and validate our models by executing some plans open-loop on a real robot-manipulator system11The video is available at https://youtu.be/JuLlliG3vGc.

We propose a novel method to tackle the visual-inertial localization problem for constrained camera movements. We use residuals from the different modalities to jointly optimize a global cost function. The residuals emerge from IMU measurements, stereoscopic feature points, and constraints on possible solutions in SE(3). In settings where dynamic disturbances are frequent, the residuals reduce the complexity of the problem and make localization feasible. We verify the advantages of our method in a suitable medical use case and produce a dataset capturing a minimally invasive surgery in the abdomen. Our novel clinical dataset MITI is comparable to state-of-the-art evaluation datasets, contains calibration and synchronization and is available at [1].

This paper proposes a quadratic programming (QP)-based variable impedance control (VIC) algorithm to solve contact-rich trajectory tracking problems with impedance, position and velocity constraints. To the best of our knowledge, the impedance constraints which are significant to ensure the worst contact compliance have never been considered in other previous works. To handle the impedance constraints of the VIC algorithm, a novel impedance model where the impedance parameters are directly served as the control input is established. The impedance-constrained VIC design problem is then formulated as a QP problem which can be efficiently solved. To handle the position and velocity constraints, a complementary force is introduced into the novel impedance model. The complementary force will appear to prevent the constraints violation when the robot approaches the constrained area. The design problem of the complementary force is also transformed into a QP problem. Combing these two QP solutions, the VIC algorithm with both impedance, position and velocity constraints can be obtained. Finally, various experiments are conducted to show the effectiveness of the proposed QP-based constrained VIC algorithm.

This paper presents a real-time control framework for consensus in operational space for robotic manipulators while satisfying task and input constraints. Consensus in operational space, as compared to joint space, enables heterogeneous robotic manipulators to achieve consensus. However, traditional frameworks tend to ignore task and input constraints while achieving consensus in operational space. We address this problem by defining safe sets in operational space and then ensure task constraint by designing Control Barrier Functions (CBF) in operational space. Control barrier functions guarantees to provide collision-free behavior for the robotic manipulator by modifying the nominal controller in a minimally invasive manner such that the trajectory of the manipulator remains in the safe set. The Quadratic Programming (QP) formulation also ensures that the nominal controller is only modified when the constraints are active, and the resulting controller is optimal in a min-norm setting. Our approach contrasts the traditional potential field method, which continues to influence the nominal controller because of its attractive and repulsive field design, and is therefore unsuitable for consensus problems. We also incorporate the input constraint in our QP formulation to ensure that the resulting controller complies with the task and input constraints. We show the efficacy of the proposed approach on 7 Degree of Freedom (DoF) KUKA LBR iiwa, 6 DoF KUKA KR5 R650 and 7 DoF Flexiv Rizon robotic manipulators, each with different dynamical and kinematic models using Dynamic Animation and Robotics Toolkit (DART) physics engine.

In this paper, we present a heterogeneous robot swarm system that can physically couple with each other to form functional structures and dynamically decouple to perform individual tasks. The connection between robots can be formed with a passive coupling mechanism, ensuring minimum energy consumption during coupling and decoupling behavior. The heterogeneity of the system enables the robots to perform structural enhancement configurations based on specific environmental requirements. We propose a connection-pair oriented configuration control algorithm to form different assemblies. We show experiments of up to nine robots performing the coupling, gap-crossing, and decoupling behaviors.

This paper considers the problem of prolonged occlusions on navigation sensors due to dust, smudges, soils, etc. Such uncontrollable occlusions often cause lower visibility as well as higher uncertainty that require considerably sophisticated behavior. To secure visibility (i.e., confidence about the world), we propose a confidence-based navigation method that encourages the robot to explore the uncertain region around the robot maximizing its local confidence. To effectively extract features from the variable size of sensor occlusions, we adopt a point-cloud based representation network. Our method returns a resilient navigation policy via deep reinforcement learning, autonomously avoiding collisions under sensor occlusions while reaching a goal. We evaluate our method in simulated and real-world environments with either static or dynamic obstacles under various sensor-occlusion scenarios. The experimental result shows that our method outperforms baseline methods under the highly occurring sensor occlusion, and achieves maximum 90% and 80% success rates in the tested static and dynamic environments, respectively.

Human motion prediction is key to understand social environments, with direct applications in robotics, surveil-lance, etc. We present a simple yet effective pedestrian trajectory prediction model aimed at pedestrians' positions prediction in urban-like environments conditioned by the environment: map and surround agents. Our model is a neural-based architecture that can run several layers of attention blocks and transformers in an iterative sequential fashion, allowing to capture the important features in the environment that improve prediction. We show that without explicit introduction of social masks, dynamical models, social pooling layers, or complicated graph-like structures, it is possible to produce on par results with SoTA models, which makes our approach easily extendable and configurable, depending on the data available. We report results performing similarly with SoTA models on publicly available and extensible-used datasets with uni-modal prediction metrics ADE and FDE.

In this work, we address the problem of solving complex collaborative robotic tasks subject to multiple varying parameters. Our approach combines simultaneous policy blending with system identification to create generalized policies that are robust to changes in system parameters. We employ a blending network whose state space relies solely on parameter estimates from a system identification technique. As a result, this blending network learns how to handle parameter changes instead of trying to learn how to solve the task for a generalized parameter set simultaneously. We demonstrate our scheme's ability on a collaborative robot and human itching task in which the human has motor impairments. We then showcase our approach's efficiency with a variety of system identification techniques when compared to standard domain randomization. The code is available on Luke Bhan's Github.

In this paper, we present an algorithm that computes funnels along trajectories of systems of ordinary differential equations. A funnel is a time-varying set of states containing the given trajectory, for which the evolution from within the set at any given time stays in the funnel. Hence it generalizes the behavior of single trajectories to sets around them, which is an important task, for example, in robot motion planning. In contrast to approaches based on sum-of-squares programming, which poorly scale to high dimensions, our approach is based on falsification and tackles the funnel computation task directly, through numerical optimization. This approach computes accurate funnel estimates far more efficiently and leaves formal verification to the end, outside all funnel size optimization loops.

Parallel kinematics machines (PKM) operate with maximal acceleration being designed for highly dynamic manipulation tasks. This leads to extreme loads of the joints, which is usually not accounted for in the motion planning. In this paper an extended inverse dynamics method is introduced, which allows computing the joint reaction forces along with the actuation torques, and provides a basis for time optimal motion planning and control minimizing wear of the components. To this end, PKM are modeled using absolute coordinates. The joint constraints are complemented with servo constraints so that the motion can be described by the actuator motion or by the end-effector motion. The presented method is particularly advantageous when certain model parameters are unknown and allows for model simplification, which would not be possible for the relative coordinate formulation. The sparsity of the obtained velocity constraint Jacobian matrix, due to the use of absolute coordinates, can be efficiently exploited to minimize computation time. The method is demonstrated and numerical results are reported for a time-optimal pick and place movement of a 4-DOF Delta robot.

Data-driven methods based on neural networks are becoming more widespread for predicting human lower-limb motion. Until now, however, actual examples have focused on only a handful, steady locomotion behaviors. Here we explore if neural network predictors can simultaneously cover many more behaviors including transient ones. Training four common types of predictor networks on a large data set of human gait, we find that they all accommodate these behaviors similarly well, maintaining prediction errors of a few centimeters (lower-limb joint positions) and degrees (joint angles) when tested on data of previously seen subjects. We further observe that although the prediction quality drops for data of unseen subjects, overall, the predicted and actual lower-limb motions remain well aligned. While the predictors demonstrated here cover the largest range of locomotion behaviors reported to date, we achieve this improvement not by better network design but simply by training on more data. This outcome clearly supports the notion that the fastest route to obtain truly general network predictors of lower-limb motion is by focusing time and effort on the rapid growth and sharing of data sets of locomotion behaviors encountered in daily life.

Assembly tasks are challenging for robot manipulation because the robot must reason over the composed effects of actions and execute multi-objective behaviors. Robots typically use predefined priorities provided by users to determine how to compose controller behaviors, but we want the robot to autonomously select these compositions based on their composed effects within the task. We present Composable Causality in Semantic Robot Programming to allow robots to reason over the composed effects of controllers when executing multi-objective actions and autonomously compose controllers without predefined priorities. Our proposed causal control basis combines controller behaviors with causal information about how the behaviors can be used to execute high-level symbolic actions. The robot uses the causal control basis to predict the transition probability of achieving the composed effects of a multi-objective action. The composed causality estimates are used to select which action to execute within the context of a furniture assembly task. We evaluate the robot's transition probability estimates in different furniture assembly trials in simulation on the Baxter robot. The robot's ability to assemble furniture using different multi-objective connection actions demonstrates the usefulness of the composed causality estimates from our causal control basis.

Robot navigation traditionally relies on building an explicit map that is used to plan collision-free trajectories to a desired target. In deformable, complex terrain, using geometric-based approaches can fail to find a path due to mischaracterizing deformable objects as rigid and impassable. Instead, we learn to predict an estimate of traversability of terrain regions and to prefer regions that are easier to navigate (e.g., short grass over small shrubs). Rather than predicting collisions, we instead regress on realized error compared to a canonical system model. We train with an on-policy approach, resulting in successful navigation policies using as little as 50 minutes of training data split across simulation and real world. Our learning-based navigation system is a sample efficient short-term planner that we demonstrate on a Clearpath Husky navigating through a variety of terrain including grassland and forest.

Foldable, origami-inspired, and laminate mechanisms are highly susceptible to deformation under external loading, which can lead to position or orientation errors if idealized kinematic models are used. According to dimensional scaling laws, laminate devices can often be treated as rigid bodies at millimeter and smaller scale deformations. However, foldable mechanisms enter the territory of soft robots at larger scales. In this paper, we show the effect of external loads applied to a laminate, 2-DOF parallel robot and the corresponding errors during a pointing task. We then present two control methods, based on deep learning, that compensates for errors caused by the material deformation in foldable robots. For each proposed control method, a Deep Neural Network (DeepNN) is trained to learn the end-effector's deformation model in no-load and loaded conditions. A DeepNN called an updating network is trained and applied in real-time using measured sensor data, in order to transfer updated weights into another DeepNN called the target network. The target network generates control signals with the aim of compensating for the end-effector's error in tracking a desired trajectory. We evaluate our proposed control methods when applied to a laminate robotic end-effector under different external loading conditions in tracking spiral paths. The experimental results show the effectiveness of our proposed control methods in compensating for material deformation in foldable robots.

Robot teleoperation via human motion tracking has been proven to be easy to learn, intuitive to operate, and facilitate faster task execution than existing baselines. However, precise control while performing the dexterous telemanipulation tasks is still a challenge. In this paper, we implement sensory augmentation in terms of haptic and augmented reality visual cues to represent four types of information critical to the precision and performance of a telemanipulation task, namely: (1) target location; (2) constraint alert; (3) grasping affordance; and (4) grasp confirmation. We further conduct two user studies to investigate the effectiveness and preferred modality of the sensory feedback against no sensory support, and how the preference will be influenced by the different types of simulated real-world additional workload. We asked 8 participants to perform a general manipulation task using a KINOVA robotic arm. Our results indicate that: (1) the haptic and AR visual cues can significantly reduce the task completion time, occurrences of errors, the total length traversed by the robot end-effector, the operational effort while increasing the interface usability; (2) the haptic feedback trended in the direction of presenting the information that needs a prompt response, while the AR visual cues are suitable to monitor the system status; (3) the participants chose their preferred feedback with the purpose of reducing the cognitive workload despite increased extra effort.

When humans control robot arms these robots often need to infer the human's desired task. Prior research on assistive teleoperation and shared autonomy explores how robots can determine the desired task based on the human's joystick inputs. In order to perform this inference the robot relies on an internal mapping between joystick inputs and discrete tasks: e.g., pressing the joystick left indicates that the human wants a plate, while pressing the joystick right indicates a cup. This approach works well after the human understands how the robot interprets their inputs - but inexperienced users still have to learn these mappings through trial and error! Here we recognize that the robot's mapping between tasks and inputs is a convention. There are multiple, equally efficient conventions that the robot could use: rather than passively waiting for the human, we introduce a shared autonomy approach where the robot actively reveals its chosen convention. Across repeated interactions the robot intervenes and exaggerates the arm's motion to demonstrate more efficient inputs while also assisting for the current task. We compare this approach to a state-of-the-art baseline - where users must identify the convention by themselves - as well as written instructions. Our user study results indicate that modifying the robot's behavior to reveal its convention outperforms the baselines and reduces the amount of time that humans spend controlling the robot. See videos of our user study here: https://youtu.be/jROTVOp469I

Industrial robots can solve tasks in controlled environments, but modern applications require robots able to operate also in unpredictable surroundings. An increasingly popular reactive policy architecture in robotics is Behavior Trees (BTs) but as other architectures, programming time drives cost and limits flexibility. The two main branches of algorithms to generate policies automatically, automated planning and machine learning, both have their own drawbacks and have not previously been combined for generation of BTs. We propose a method for creating BTs by combining these branches, inserting the result of an automated planner into the population of a Genetic Programming algorithm. Experiments confirm that the proposed method performs well on a variety of robotic assembly problems and outperforms the base methods used separately. We also show that this high level learning of Behavior Trees can be transferred to a real system without further training.