IROS 2021 Papers

There is a growing interest in designing robots that can work alongside humans. Such robots will undoubtedly be expected to explain their behavior and decisions. While generating explanations is an actively researched topic, most works tend to focus on methods that generate explanations that are one size fits all. As in the specifics of the user-model are completely ignored. The handful of works that look at tailoring their explanation to the user’s background rely on having specific models of the users (either analytic models or learned labeling models). The goal of this work is thus to propose an end-to-end adaptive explanation generation system that begins by learning the different types of users that the robot could interact with. Then during the interaction with the target user, it is tasked with identifying the type on the fly and adjust its explanations accordingly. The former is achieved by a data-driven clustering approach while for the latter, we compile our explanation generation problem into a POMDP. We demonstrate the usefulness of our system on two domains using state-of-the-art POMDP solvers. We also report the results of a user study that investigates the benefits of providing personalized explanations in a human-robot interaction setting.

3D skeleton-based motion prediction and activity recognition are two interwoven tasks in human behaviour analysis. In this work, we propose a motion context modeling methodology that provides a new way to combine the advantages of both graph convolutional neural networks and recurrent neural networks for joint human motion prediction and activity recognition. Our approach is based on using an LSTM encoder-decoder and a non-local feature extraction attention mechanism to model the spatial correlation of human skeleton data and temporal correlation among motion frames. The proposed network can easily include two output branches, one for Activity Recognition and one for Future Motion Prediction, which can be jointly trained for enhanced performance. Experimental results on Human 3.6M, CMU Mocap and NTU RGB-D datasets show that our proposed approach provides the best prediction capability among baseline LSTM-based methods, while achieving comparable performance to other state-of-the-art methods.

Transporting objects using quadrotors with cables has been widely studied in the literature. However, most of those approaches assume that the cables are previously attached to the load by human intervention. In tasks where multiple objects need to be moved, the efficiency of the robotic system is constrained by the requirement of manual labor. Our approach uses a non-stretchable cable connected to two quadrotors, which we call the catenary robot, that fully automates the transportation task. Using the cable, we can roll and drag the cuboid object (box) on planar surfaces. Depending on the surface type, we choose the proper action, dragging for low friction, and rolling for high friction. Therefore, the transportation process does not require any human intervention as we use the cable to interact with the box without requiring fastening. We validate our control design in simulation and with actual robots, where we show them rolling and dragging boxes to track desired trajectories.

Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing locomotion capability in typical indoor scenarios, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR HMD worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying a synthesized image of the operator with facial animation, which enables direct interaction with remote persons. We evaluate our system successfully both in a user study with untrained operators as well as a longer and more complex integrated mission. We discuss lessons learned from the trials and possible improvements.

Precise perception is one of the key enablers of autonomous robotic operations. The right selection of sensors significantly influences the overall performance of the system. This paper provides a systematic approach for evaluation of various sensors available on the market. The main focus is to assess the performance in use cases of short to medium distance operations, especially relevant for precise manipulation and/or quality control. The evaluation is based solely on depth data (point clouds). We use six metrics to evaluate the sensors and propose a novel approach for low-cost fabrication of benchmark targets. The evaluation experiments are conducted on different materials to simulate various industrial environments. Our results provide a qualitative and quantitative comparison of different characteristics of various sensors and can be used to select an appropriate device for specific conditions.

In this work, we present a definition of a neurointerface architecture combined from two main parts (1) neuroport (a hardware device) that implements a neuro protocol, generated and managed by a (2) neuroterminal (a software). The proposed architecture was created by analogy with OSI network architecture. We also present the neuroterminal as an oscillator motif real-time neurosimulation and results of the comparison of a bio-plausible motor pattern generated by oscillator motifs with square pulses of 20 – 40 Hz used as the neuro protocol for the output neuroport and measured their discomfort rate and efficacy according to an angle of subject fingers deflection. We determined that the most effective is the five oscillator motifs generated pattern for a median nerve stimulation, whereas for a muscle simulation 20 and 40 Hz are more effective. We indicate that the oscillator motif generated pattern feels more natural than square pulses 20 – 40 Hz, which feel like a spasm.

In-flight objects capture is extremely challenging. The robot is required to complete trajectory prediction, interception position calculation and motion planning within tens of milliseconds. As in-flight uneven objects are affected by various kinds of forces, which leads to the time-varying acceleration, motion prediction for them is difficult. In order to compensate the system’s non-linearity, we propose using a recurrent neural network model, which we call the Neural Acceleration Estimator (NAE), to estimate the varying acceleration by observing a small fragment of previous deflected trajectory without any prior information. Moreover, end-to-end training with Differantiable Filter (NAE-DF) gives a supervision for measurement uncertainty and further improves the prediction accuracy. Experimental results show that motion prediction with NAE and NAE-DF is superior to other methods and has a good generalization performance on unseen objects. We test our methods on a robot, performing velocity control in real world and respectively achieve 83.3% and 86.7% success rate on a ploy urethane banana and a gourd. We also release an object in-flight dataset containing 1,500 trajectorys for uneven objects, which can be found on the project website:https://sites.google.com/view/neural-motion-prediction.

The advent of deep learning has inspired research into end-to-end learning for a variety of problem domains in robotics. For navigation, the resulting methods may not have the generalization properties desired let alone match the performance of traditional methods. Instead of learning a navigation policy, we explore learning an adaptive policy in the parameter space of an existing navigation module. Having adaptive parameters provides the navigation module with a family of policies that can be dynamically reconfigured based on the local scene structure and addresses the common assertion in machine learning that engineered solutions are inflexible. Of the methods tested, reinforcement learning (RL) is shown to provide a significant performance boost to a modern navigation method through reduced sensitivity of its success rate to environmental clutter. The outcomes indicate that RL as a meta-policy learner, or dynamic parameter tuner, effectively robustifies algorithms sensitive to external, measurable nuisance factors.

We present a new neural network architecture, called NaturalNet, which uses a simplified biological neuron model and consists of a set of nonlinear ordinary differential equations. We model the membrane potential of each neuron by integrating the in-flowing currents, but we do not consider ion channels, nor individual spikes. To keep the membrane potential within a defined value range, we introduce a suitable clipping mechanism. With our approach, we aim to develop agents solving complex tasks by providing a higher biological plausibility than commonly used neural networks for deep learning applications, while also offering low computational complexity to enable fast training. To demonstrate the learning capabilities of NaturalNets, we use the virtual robotic environments of the OpenAI Gym framework, a widely-used toolkit for developing and comparing reinforcement learning algorithms. We compared a variety of different widespread neural network architectures, including long short-term memory (LSTM), gated recurrent units (GRUs), feedforward, and Elman networks. Our experiments show that NaturalNets were able to perform well for all considered virtual robotic control tasks, where we apply the covariance matrix adaptation evolutionary strategy (CMAES) for training.

Visual place recognition (VPR) is critical in not only localization and mapping for autonomous driving vehicles, but also assistive navigation for the visually impaired population. To enable a long-term VPR system on a large scale, several challenges need to be addressed. First, different applications could require different image view directions, such as front views for self-driving cars while side views for the low vision people. Second, VPR in metropolitan scenes can often cause privacy concerns due to the imaging of pedestrian and vehicle identity information, calling for the need for data anonymization before VPR queries and database construction. Both factors could lead to VPR performance variations that are not well understood yet. To study their influences, we present the NYU-VPR dataset that contains more than 200,000 images over a 2km×2km area near the New York University campus, taken within the whole year of 2016. We present benchmark results on several popular VPR algorithms showing that side views are significantly more challenging for current VPR methods while the influence of data anonymization is almost negligible, together with our hypothetical explanations and in-depth analysis.

Motion control and planning for the manipulator are critical components in manipulator teleoperation. Online (real-time) motion control is challenging for active obstacle avoidance and often results in fluctuating and unsafe motion. Offline motion planning, on the other hand, generates precise and secure trajectories for complex manipulation. In this paper, a real-time nonlinear model predictive control based motion planner (NMPC-MP) is designed for teleoperated manipulation. In contrast to traditional NMPC-based approaches, our model considers a complex environment with dynamic obstacles. Our multi-threaded NMPC-MP allows for real-time planning, including dynamic objects. We evaluate our approach both in a simulated environment and with real-world experiments using the Kinova® Movo platform. The comparison to state-of-the-art approaches (e.g., RRT-Connect, CHOMP, and STOMP) shows a significant improvement in real-time motion planning using NMPC-MP. In real-world tests, the proposed planner was applied on a human-shaped dual manipulator setup. Our results show that the NMPC-MP runs in real-time and generates smooth and reliable trajectories. The experiments validate that the planner is able to precisely track active goals from the teleoperator while avoiding self-collision and obstacles.

Prior experiments have shown that human gait synchronizes to periodic torque pulses applied about the hip and ankle joints by robotic exoskeletons. Importantly, entrainment occurred even when the pulse period differed slightly from the user’s preferred stride period, making it a viable approach to increase gait speed. As gait speed is an important outcome of gait therapy, gait entrainment to mechanical perturbations may serve as a promising new method of robot-aided therapy. Still, an understanding of the underlying neuromechanical processes that give rise to gait entrainment is needed to fully evaluate its therapeutic potential. To gain such insight, the goal of this paper was to evaluate whether an existing neuromechanical model of human locomotion exhibited entrainment behavior similar to that observed in the prior human experiments. Simulation results showed that the model entrained to pulses applied at both the ankle and hip joints. The convergence of relative phase between model gait and hip perturbations was similar to that observed with the human gait, but differed slightly for ankle perturbations. Thus, models that can more accurately describe neuromechanical interactions between human gait and robotic exoskeletons are still needed. Nevertheless, the simulation results support the notion that the limit-cycle behavior observed during locomotion does not require supra-spinal control or a self-sustaining oscillatory neural network, which has important implications for improving gait therapy.

There is an increasing demand for accurate prediction of joint moments using wearable sensors for robotic exoskeletons to achieve precise control and for rehabilitation care to remotely monitor users’ condition. In this study, we used electromyography (EMG) signals to first identify muscle synergies, then used them to train of a long short-term memory network to predict knee joint moments during walking. Kinematics, ground reaction forces and EMG from 10 muscles on the right limb were collected from 6 able-bodied subjects during normal gait. Between 4 and 6 muscle synergies were extracted from the EMG signals, generating two outputs - the muscle synergies weight matrix and the time-dependent muscle synergies action signals. The muscle synergies action signals and measured knee joint moments from inverse dynamics were then used as inputs to train the joint moment prediction model using a long short-term memory network. For testing, between 4 and 7 EMG signals were used to estimate the muscle synergies action signals with the extracted muscle synergies weights matrix. The estimated muscle synergies action signals were then used to predict knee joint moments. Knee joint moments were also predicted directly from all 10 EMGs, then from 4-7 EMG signals using another long short-term memory network. Prediction accuracy from the synergies-trained network vs. the EMG-trained network were compared, using the same number of EMG signals in each. Prediction error with respect to moments measured via inverse dynamics was computed for both networks. Knee moments predicted with as few as 4 EMGs was at least as accurate as moments predicted from all 10 EMGs when muscle synergies were exploited. Predicted knee moments from muscle synergies achieved an average of 4.63% root mean square error from 4 EMG signals, which was lower than error when predicted directly from 4 EMG signals (5.63%).

When a learning solution is needed for different robots, a model is often trained for each robot geometry, even if the robotic task is the same and the robots are structurally similar. In this paper, we address the problem of transfer learning of swept volume predictors for the motion of articulated robots with similar geometric structure. The swept volume is a scalar value corresponding to the space occupied by an entire motion of the robot. Swept volume has many applications, including being an ideal distance measure for sampling based motion planners, but it is expensive to compute. We address this learning problem through a multitask network where a common input is used to learn multiple related tasks. In this work a single network learns the kinematic-geometric information common among robots. In order to identify the properties of our multitask network favorable for transfer, we evaluate transfer properties of several shared layers, number of robots in multitask training, and feature layers. We demonstrate positive transfer results with a training set that is a fraction of the data size used in the multitask and baseline training. All the robots considered are 7-DOF manipulators with links with a variety of lengths and shapes. We also present a study of the weights and activations of the trained networks that show high correlation with the transferability patterns we observed.

Assistive robots that operate alongside humans require the ability to understand and replicate human behaviours during a handover. A handover is defined as a joint action between two participants in which a giver hands an object over to the receiver. In this paper, we present a method for learning human-to-human handovers observed from motion capture data. Given the giver and receiver pose from a single timestep, and the object label in the form of a word embedding, our Multitask Variational Autoencoder jointly forecasts their pose as well as the orientation of the object held by the giver at handover. Our method is in large contrast to existing works for human pose forecasting that employ deep autoregressive models requiring a sequence of inputs. Furthermore, our method is novel in that it learns both the human pose and object orientation in a joint manner. Experimental results on the publicly available Handover Orientation and Motion Capture Dataset show that our proposed method outperforms the autoregressive baselines for handover pose forecasting by approximately 20% while being on-par for object orientation prediction with a runtime that is 5x faster. a

Active inference, a theoretical construct inspired by brain processing, is a promising alternative to control artificial agents. However, current methods do not yet scale to high-dimensional inputs in continuous control. Here we present a novel active inference torque controller for industrial arms that maintains the adaptive characteristics of previous proprioceptive approaches but also enables large-scale multimodal integration (e.g., raw images). We extended our previous mathematical formulation by including multimodal state representation learning using a linearly coupled multimodal variational autoencoder. We evaluated our model on a simulated 7DOF Franka Emika Panda robot arm and compared its behavior with a previous active inference baseline and the Panda built-in optimized controller. Results showed improved tracking and control in goal-directed reaching due to the increased representation power, high robustness to noise and adaptability in changes on the environmental conditions and robot parameters without the need to relearn the generative models nor parameters retuning.

Soft robotic gloves have shown great advantages in assisting individuals with hand pathologies to perform continuous exercises to restore their hand functions, which could considerably accelerate the rehabilitation process and reduce the costs. However, single rehabilitation mode, difficulty in achieving multiple degrees-of-freedom (DoF) motion, and the lack of high-fidelity feedback still challenge the development of soft robotic gloves. In this paper, we propose a novel design of a robotic glove based on soft-rigid hybrid joint actuators and minimal clutches. We first introduce structures and working principles of the proposed bending joint actuator in detail and then characterize the single joint actuator. Furthermore, we present a performance evaluation of the whole robotic glove in both active and passive modes. Preliminary experimental results showed that (1) in the active training mode, the tested human hand’s muscle effort needed to conduct gross finger flexion increased from 11.16% to 42.60% of the maximum value when the air pressure inside the minimal clutches changed from 0 kPa to 200 kPa; (2) in the passive mode, the 10-DoF robotic glove could assist the tested hand to perform various training exercises and grasp various objects with different hand postures. This paper focuses on the integrated design of multi-DoF structures and variable stiffness mechanisms, which will have an impact on the development of multifunctional soft robots and wearable devices.

Offroad mobile robot perception systems must be able to learn robust terrain classification models. Models built from computer vision often fail in their ability to generalize to new environments where appearance characteristics change. Sound and vibration signals from robot-terrain interaction can be used to classify the terrain from characteristics that vary less between environments. Previous work using sound and vibration for terrain classification has only classified ground terrain types. We extend here to building a 7-class multiclass classifier that can classify both ground and above-ground terrain types in challenging outdoor off-road settings, thereby increasing the semantic richness of the terrain classification. Our contributions include: 1) We instrument a robotic vehicle with a variety of sound and vibration sensors mounted at different vehicle locations and directions, as well as color cameras. 2) We collect interactive and visual field data from many outdoor off-road sites with different environments. 3) We build multiclass classifiers for different combinations of sound and vibration signals, and we autonomously learn the optimal signal combination. We compare this against a single microphone from our previous work [1]. 4) We benchmark both of these results against a state-of-the art vision system. All of these multiclass classifiers are tested at different locations from where they are trained. By using one microphone instead of the vision system, we increase balanced accuracy from 70% to 82%. By using the optimal sound and vibration combination, we increase balanced accuracy from 82% to 87%. All four of these contributions are field robotics in nature: we build a sensor system and then we use that system to collect new field data that allows for a comparative evaluation of different modules of the system. Such datasets do not exist that include these varying sensors on varying field terrain. We are also contributing to machine learning research by ... Show More

We present a system for multi-level scene awareness for robotic manipulation. Given a sequence of camera-inhand RGB images, the system calculates three types of information: 1) a point cloud representation of all the surfaces in the scene, for the purpose of obstacle avoidance. 2) the rough pose of unknown objects from categories corresponding to primitive shapes (e.g., cuboids and cylinders), and 3) full 6-DoF pose of known objects. By developing and fusing recent techniques in these domains, we provide a rich scene representation for robot awareness. We demonstrate the importance of each of these modules, their complementary nature, and the potential benefits of the system in the context of robotic manipulation.