IROS 2023 Papers

This paper presents a novel method for model-free prediction of grasp poses for suction grippers with multiple suction cups. Our approach is agnostic to the design of the gripper and does not require gripper-specific training data. In particular, we propose a two-step approach, where first, a neural network predicts pixel-wise grasp quality for an input image to indicate areas that are generally graspable. Second, an optimization step determines the optimal gripper selection and corresponding grasp poses based on configured gripper layouts and activation schemes. In addition, we introduce a method for automated labeling for supervised training of the grasp quality network. Experimental evaluations on a real-world industrial application with bin picking scenes of varying difficulty demonstrate the effectiveness of our method.

Terrestrial and aerial bimodal vehicles have gained widespread attention due to their cross-domain maneuverability. Nevertheless, their bimodal dynamics significantly increase the complexity of motion planning and control, thus hindering robust and efficient autonomous navigation in unknown environments. To resolve this issue, we develop a model-based planning and control framework for terrestrial aerial bi-modal vehicles. This work begins by deriving a unified dynamic model and the corresponding differential flatness. Leveraging differential flatness, an optimization-based trajectory planner is proposed, which takes into account both solution quality and computational efficiency. Moreover, we design a tracking controller using nonlinear model predictive control based on the proposed unified dynamic model to achieve accurate trajectory tracking and smooth mode transition. We validate our framework through extensive benchmark comparisons and experiments, demonstrating its effectiveness in terms of planning quality and control performance.

This paper is concerned with the issue of the kinematic model-based bending control for the magnetically actuated robotic endoscope and its application for automatic retroflexion. By the utilization of the Cosserat rod theory and the transformation in the magnetic tip of the endoscope, the comprehensive kinematic model of the magnetically-actuated robotic endoscope is established. Afterward, a magnetic control scheme for the bending motion is proposed by co-developing an error feedback PID control strategy and the model-based feedback approach. Moreover, as one unique kind of bending motion, retroflexion is taken into account, and the strategy aimed at the bid of compact space retroflexion is presented by virtue of the introduction of serial waypoints pursuing the position of the magnetic tip being close to the midline as possible. Eventually, the developed modeling and bending control scheme and the compact space retroflexion strategy are examined in a magnetically actuated robotic endoscope system to manifest the effectiveness and applicability of the theoretical approach. The experimental results indicate that the designed controller can drive the endoscope to bend to the desired pose and show a reduction of about 47.01% in the sweeping area and 79.25% in the last distance to midline achieved by conducting compact space retroflexion in comparison to “U” type one.

Emotion detection presents challenges to intelligent human-robot interaction (HRI). Foundational deep learning techniques used in emotion detection are limited by information-constrained datasets or models that lack the necessary complexity to learn interactions between input data elements, such as the the variance of human emotions across different contexts. In the current effort, we introduce 1) MoEmo (Motion to Emotion), a cross-attention vision transformer (ViT) for human emotion detection within robotics systems based on 3D human pose estimations across various contexts, and 2) a data set that offers full-body videos of human movement and corresponding emotion labels based on human gestures and environmental contexts. Compared to existing approaches, our method effectively leverages the subtle connections between movement vectors of gestures and environmental contexts through the use of cross-attention on the extracted movement vectors of full-body human gestures/poses and feature maps of environmental contexts. We implement a cross-attention fusion model to combine movement vectors and environment contexts into a joint representation to derive emotion estimation. Leveraging our Naturalistic Motion Database, we train the MoEmo system to jointly analyze motion and context, yielding emotion detection that outperforms the current state-of-the-art.

This study presents a new hardware design and control of a minimally actuated 5 control degrees of freedom (CDoF) quadrotor-based tiltrotor. The proposed tiltrotor possesses several characteristics distinct from those found in existing works, including: 1) minimal number of actuators for 5 CDoF, 2) large margin to generate interaction force during aerial physical interaction (APhI), and 3) no mechanical obstruction in thrust direction rotation. Thanks to these properties, the proposed tiltrotor is suitable for perching-enabled APhI since it can hover parallel to an arbitrarily oriented surface and can freely adjust its thrust direction. To fully control the 5-CDoF of the designed tiltrotor, we construct an asymptotically stabilizing controller with stability analysis. The proposed tiltrotor design and controller are validated in experiments where the first two experiments of x, y position tracking and pitch tracking show controllability of the added CDoF compared to a conventional quadrotor. Finally, the last experiment of perching and cart pushing demonstrates the proposed tiltrotor's applicability to perching-enabled APhI.

Collective perception is a foundational problem in swarm robotics, in which the swarm must reach consensus on a coherent representation of the environment. An important variant of collective perception casts it as a best-of-n decision-making process, in which the swarm must identify the most likely representation out of a set of alternatives. Past work on this variant primarily focused on characterizing how different algorithms navigate the speed-vs-accuracy tradeoff in a scenario where the swarm must decide on the most frequent environmental feature. Crucially, past work on best-of-n decision-making assumes the robot sensors to be perfect (noise- and fault-less), limiting the real-world applicability of these algorithms. In this paper, we apply optimal estimation techniques and a decentralized Kalman filter to derive, from first principles, a probabilistic framework for minimalistic swarm robots equipped with flawed sensors. Then, we validate our approach in a scenario where the swarm collectively decides the frequency of a certain environmental feature. We study the speed and accuracy of the decision-making process with respect to several parameters of interest. Our approach can provide timely and accurate frequency estimates even in presence of severe sensory noise.

Failure of any component in a robotic system during operation is a critical concern, and it is essential to address such incidents promptly. This work investigates a novel technique to recover from failures or changes in the configuration space while avoiding expensive re-computation or re-planning. We propose the Minimal Path Violation (MPV) concept to find the best feasible path with minimal re-configurations. The algorithm ranks pathways based on visibility, expansiveness, and cost. We perform experiments with articulated 3 DOF to 28 DOF robots ranging from serial linkage robots, Kuka YouBots, and PR2 robots. Our results show that our method outperforms existing optimal planners in computation time, total nodes, and path cost while preserving path feasibility in changed configuration space.

The untethered microrobots driven by multiple external physics fields have promising ability in minimally invasive disease treatments. One common type of the driving fields is gradient magnetic field, which can provide microrobots with adequate driving force in complicated environment. In this study, a control method of microrobot through gradient magnetic field system is presented, which is realized by moving the field free point (FFP) to produce an alterable magnetic driving force. A confirmatory experiment of the robot reciprocating motion control is undertaken in a 1D gradient magnetic robot system. The control method could be applied to further studies on in vivo applications of targeted microrobot drug delivery system.

Studies of the human brain during natural activities, such as locomotion, would benefit from the ability to image deep brain structures during these activities. While Positron Emission Tomography (PET) can image these structures, the bulk and weight of current scanners are not compatible with the desire for a wearable device. This has motivated the design of a robotic system to support a PET imaging system around the subject's head and to move the system to accommodate natural motion. We report here the design and experimental evaluation of a prototype robotic system that senses motion of a subject's head, using parallel string encoders connected between the robot-supported imaging ring and a helmet worn by the subject. This measurement is used to robotically move the imaging ring (coarse motion correction) and to compensate for residual motion during image reconstruction (fine motion correction). Minimization of latency and measurement error are the key design goals, respectively, for coarse and fine motion correction. The system is evaluated using recorded human head motions during locomotion, with a mock imaging system consisting of lasers and cameras, and is shown to provide an overall system latency of about 80 ms, which is sufficient for coarse motion correction and collision avoidance, as well as a measurement accuracy of about 0.5 mm for fine motion correction.

This paper introduces a novel approach to video object detection detection and tracking on Unmanned Aerial Vehicles (UAVs). By incorporating metadata, the proposed approach creates a memory map of object locations in actual world coordinates, providing a more robust and interpretable representation of object locations in both, image space and the real world. We use this representation to boost confidences, resulting in improved performance for several temporal computer vision tasks, such as video object detection, short and long-term single and multi-object tracking, and video anomaly detection. These findings confirm the benefits of metadata in enhancing the capabilities of UAVs in the field of temporal computer vision and pave the way for further advancements in this area.

To personalize the robot guide experience, the robot needs to detect a person's indifference and adjust its explanation toward the person's interest in topics. However, detecting the person's indifference is challenging in a museum, as we cannot use a bulky wearable or facial expression recognition due to unexpected light condition or standing position. We propose to observe people's behaviors and movements on detecting people's indifference. To prove its feasibility, we invited 11 participants to our in-lab museum-like environment. Our robot explains exhibits while videorecording the interaction. Then, we asked participants to watch the recordings and report when they felt bored or indifferent to the explanation. We labelled their movement and matched them to their report so that we know which behaviors and movements hint the person's indifference. We used the decision tree and random forest methods to understand the common pattern when people are indifferent during the explanation in a museum scenario. From our observation experiment, we found that if the listener nods their heads many times or looks at the exhibit for a long time, they are likely interested in the topic, fewer overall movements or looking elsewhere hint that the listener may be indifferent, and if the explanation goes longer than three minutes, the listener would be likely bored.

During a natural disaster such as hurricane, earthquake, or fire, robots have the potential to explore vast areas and provide valuable aid in search & rescue efforts. These scenarios are often high-pressure and time-critical with dynamically-changing task goals. One limitation to these large scale deployments is effective human-robot interaction. Prior work shows that collaboration between one human and one robot benefits from shared control. Here we evaluate the efficacy of shared control for human-swarm teaming in an immersive virtual reality environment. Although there are many human-swarm interaction paradigms, few are evaluated in high-pressure settings representative of their intended end use. We have developed an open-source virtual reality testbed for realistic evaluation of human-swarm teaming performance under pressure. We conduct a user study (\mathrm{n}=16\mathrm{n}=16) comparing four human-swarm paradigms to a baseline condition with no robotic assistance. Shared control significantly reduces the number of instructions needed to operate the robots. While shared control leads to marginally improved team performance in experienced participants, novices perform best when the robots are fully autonomous. Our experimental results suggest that in immersive, high-pressure settings, the benefits of robotic assistance may depend on how the human and robots interact, and the human operator's expertise.

This paper presents the underactuated vacuum gripper named Polypus. What is unique in Polypus is that it combines under-actuation and vacuum grasping to apply both power and unilateral grasp to objects of various shape and geometry. In addition, the gripper features modularity, i.e., single phalanges can be added or removed based on the application. The high flexibility also comes with a cost-effective (less than 100€ and simple design that can be manufactured with a consumer-grade 3D printer using FDM technology. While the analytical model has been introduced by the authors in previous research, here its experimental validation is described using a physical prototype that shows that the theoretical assumptions are reasonable. Experimental results also suggest a small variation in the original theoretical model.

From construction materials, such as sand or asphalt, to kitchen ingredients, like rice, sugar, or salt; the world is full of granular materials. Despite impressive progress in robotic manipulation of single objects, granular materials remain a challenge due to difficulties in modelling these highly deformable and inhomogeneous materials, which are governed by dynamics that are hard to capture analytically. We argue that despite the high degrees of freedom and the complex underlying dynamics of granular materials, many practical problems that require manipulating them can be solved by leveraging simple models, informative motion priors, and a fast feedback loop. In this work, we show that computational Optimal Transport (OT) can be leveraged to derive informative, robot-agnostic motion priors for transforming a pile of granular materials from a source into a target distribution and generate robot motion plans with a next-best sweep planner that uses a simple material-agnostic sweep model. We plan sweeps directly on a height map representation of the material distribution and hence avoid a costly particle-level treatment of the problem. We validate our approach with a large set of simulation and hardware experiments that demonstrate several complex shaping tasks, including gathering, separating, and writing letters with different types of granular materials.

Multimodal demonstrations provide robots with an abundance of information to make sense of the world. However, such abundance may not always lead to good performance when it comes to learning sensorimotor control policies from human demonstrations. Extraneous data modalities can lead to state over-specification, where the state contains modalities that are not only useless for decision-making but also can change data distribution across environments. State over-specification leads to issues such as the learned policy not generalizing outside of the training data distribution. In this work, we propose Masked Imitation Learning (MIL) to address state over-specification by selectively using informative modalities. Specifically, we design a masked policy network with a binary mask to block certain modalities. We develop a bi-level optimization algorithm that learns this mask to accurately filter over-specified modalities. We demonstrate empirically that MIL outperforms baseline algorithms in simulated domains and effectively recovers the environment-invariant modalities on a multimodal dataset collected on a real robot. Videos and supplemental details are at: https://tinyurl.com/masked-il

Recent works in object detection in LiDAR point clouds mostly focus on predicting bounding boxes around objects. This prediction is commonly achieved using anchor-based or anchor-free detectors that predict bounding boxes, requiring significant explicit prior knowledge about the objects to work properly. To remedy these limitations, we propose MaskBEV, a bird's-eye view (BEV) mask-based object detector neural architecture. MaskBEV predicts a set of BEV instance masks that represent the footprints of detected objects. Moreover, our approach allows object detection and footprint completion in a single pass. MaskBEV also reformulates the detection problem purely in terms of classification, doing away with regression usually done to predict bounding boxes. We evaluate the performance of MaskBEV on both SemanticKITTI and KITTI datasets while analyzing the architecture advantages and limitations.

Fiducial markers can encode rich information about the environment and aid Visual SLAM (VSLAM) approaches in reconstructing maps with practical semantic information. Current marker-based VSLAM approaches mainly utilize markers for improving feature detections in low-feature environments and/or incorporating loop closure constraints, generating only low-level geometric maps of the environment prone to inaccuracies in complex environments. To bridge this gap, this paper presents a VSLAM approach utilizing a monocular camera along with fiducial markers to generate hierarchical representations of the environment while improving the camera pose estimate. The proposed approach detects semantic entities from the surroundings, including walls, corridors, and rooms encoded within markers, and appropriately adds topological constraints among them. Experimental results on a real-world dataset collected with a robot demonstrate that the proposed approach outperforms a marker-based VSLAM baseline in terms of accuracy, given the addition of new constraints while creating enhanced map representations. Furthermore, it shows satisfactory results when comparing the reconstructed map quality to the one rebuilt using a LiDAR SLAM approach.

Reliable monitoring of vessel motions is crucial for safe and efficient operation of marine vessels. Pitching and rolling motions are commonly monitored using high-grade inertial measurement units (IMUs). However, such sensors become unreliable in presence of long-lasting accelerations. In this work, we propose a method for attitude estimation of marine vessels relying on an image stream and known world features. Our focus is on the estimation of pitch and roll angles. We employ a semantic segmentation network and process its output for robust extraction of coastlines and horizon. The image features are matched with known world features to estimate the attitude. The proposed method is validated using different metrics on data acquired from a small passenger ferry. The proposed method achieves more than 60% reduction in vertical reprojection error compared to IMU. We show that the proposed method outperforms IMU and can be used to replace it whenever horizon or coastline is visible.

When faced with an unstructured environment filled with an unknown number and size of obstacles on a chaotic terrain, it can be a challenge to determine the best method for navigating and mapping the space. This problem, known as Simultaneous Localization and Mapping (SLAM), has typically been approached using vision-based solutions, but these solutions require clear visual conditions in order to function optimally. A different approach to sensing environments has been explored in soft robotic systems, specifically by sensing changes in the environment through sensing changes in the robot's configuration. Building on this idea, we introduce a method of mapping based on colliding with and deforming around obstacles using a soft, growing robot. Instead of avoiding obstacles, as is typically done to protect robots, we take advantage of the soft, growing robot's compliance in order to navigate through, and collect information about, the environment. Through the construction and testing of a geometry-based simulation, we analyzed the behavior and ef-fectiveness of this approach for mapping by generating random launch positions and collecting information from contacted obstacles and traversed regions. Through a myriad of randomly generated environments, we determine that: 1) the density of obstacles in an environment has minimal impact on mapping abilities and 2) at least 70% of each environment tested can be mapped by deploying 20 or fewer soft, growing robots.