IROS 2023 Papers

When robots retrieve specific objects from cluttered scenes, such as home and warehouse environments, the target objects are often partially occluded or completely hidden. Robots are thus required to search, identify a target object, and successfully grasp it. Preceding works have relied on pre-trained object recognition or segmentation models to find the target object. However, such methods require laborious manual annotations to train the models and even fail to find novel target objects. In this paper, we propose an Image-driven Object Searching and Grasping (IOSG) approach where a robot is provided with the reference image of a novel target object and tasked to find and retrieve it. We design a Target Similarity Network that generates a probability map to infer the location of the novel target. IOSG learns a hierarchical policy; the high-level policy predicts the subtask type, whereas the low-level policies, explorer and coordinator, generate effective push and grasp actions. The explorer is responsible for searching the target object when it is hidden or occluded by other objects. Once the target object is found, the coordinator conducts target-oriented pushing and grasping to retrieve the target from the clutter. The proposed pipeline is trained with full self-supervision in simulation and applied to a real environment. Our model achieves a 96.0% and 94.5% task success rate on coordination and exploration tasks in simulation respectively, and 85.0% success rate on a real robot for the search-and-grasp task. Please refer to our project page for more information: https://z.umn.edu/iosg.

Sensor fusion has become a popular topic in robotics. However, conventional fusion methods encounter many difficulties, such as data representation differences, sensor variations, and extrinsic calibration. For example, the calibration methods used for LiDAR-camera fusion often require manual operation and auxiliary calibration targets. Implicit neural representations (INRs) have been developed for 3D scenes, and the volume density distribution involved in an INR unifies the scene information obtained by different types of sensors. Therefore, we propose implicit neural fusion (INF) for LiDAR and camera. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field. Along with the training process, INF both estimates LiDAR poses and optimizes extrinsic parameters. Our experiments demonstrate the high accuracy and stable performance of the proposed method.

In this paper, we tackle the control and trajectory planning problems for the cooperative transportation system of cable-suspended payload with multi Unmanned Aerial Vehicles (UAVs). Firstly, a payload controller is presented considering the dynamic coupling between the UAV and the payload to accomplish the active suppression of payload swing and the complex payload trajectory tracking. Secondly, different from the simplification of obstacles in most approaches, we propose three Insetting Formation (IF) algorithms for the complete obstacle shape to generate collision-free waypoints for the cooperative transportation system. An IF strategy is proposed by integrating three IF algorithms to improve the success rate of obstacle avoidance and reduce the algorithm complexity for performing the aggressive flight. Finally, we verify the robustness and high performance of the proposed algorithm through benchmark comparison and real-world experiments. Moreover, our source code is released as an open-source ros package.

Mixup-based data augmentation has been validated to be a critical stage in the self-training framework for unsupervised domain adaptive semantic segmentation (UDASS), which aims to transfer knowledge from a well-annotated (source) domain to an unlabeled (target) domain. Existing self-training methods usually adopt the popular region-based mixup techniques with a random sampling strategy, which unfortunately ignores the dynamic evolution of different semantics across various domains as training proceeds. To improve the UDA-SS performance, we propose an Informed Domain Adaptation (IDA) model, a self-training framework that mixes the data based on class-level segmentation performance, which aims to emphasize small-region semantics during mixup. In our IDA model, the class-level performance is tracked by an expected confidence score (ECS). We then use a dynamic schedule to determine the mixing ratio for data in different domains. Extensive experimental results reveal that our proposed method is able to outperform the state-of-the-art UDA-SS method by a margin of 1.1 mIoU in the adaptation of GTA-V to Cityscapes and of 0.9 mIoU in the adaptation of SYNTHIA to Cityscapes. Code link: https://github.com/ArlenCHEN/IDA.git

3D object detection have achieved significant performance in many fields, e.g., robotics system, autonomous driving, and augmented reality. However, most existing methods could cause catastrophic forgetting of old classes when performing on the class-incremental scenarios. Meanwhile, the current class-incremental 3D object detection methods neglect the relationships between the object localization information and category semantic information, and assume all the knowledge of old model is reliable. To address the above challenge, we present a novel Incremental 3D Object Detection framework with the guidance of prompting, i.e., I3DOD. Specifically, we propose a task-shared prompts mechanism to learn the matching relationships between the object localization information and category semantic information. After training on the current task, these prompts will be stored in our prompt pool, and perform the relationship of old classes in the next task. Moreover, we design a reliable distillation strategy to transfer knowledge from two aspects: a reliable dynamic distillation is developed to filter out the negative knowledge and transfer the reliable 3D knowledge to new detection model; the relation feature is proposed to capture the responses relation in feature space and protect plasticity of the model when learning novel 3D classes. To the end, we conduct comprehensive experiments on two benchmark datasets and our method outperforms the state-of-the-art object detection methods by 0.6% ∼ 2.7% in terms of mAP@0.25.

Sophisticated manipulation requires both compliance and accuracy. While tactile robots excel at being compliant, their accuracy is often inadequate for complex manipulation. Contact-rich assembly tasks, such as the insertion and manipulation of objects with small tolerances pose an enormous challenge. Complex, highly integrated assemblies, especially in high-tech areas such as robotics, sensors, or machines, still require human personnel, as they cannot be automated in a satisfactory way. To automate such tasks, especially in the context of labor shortage and Industry 4.0, these limitations must be overcome. Robots need to guarantee force limits for active environments in order to avoid harm or damage. Therefore, in this work, we adapt standard Cartesian impedance control by introducing an integration term for position accuracy and wrench limits for safe compliant interaction with unknown and active environments. We combine this with a virtual energy tank to guarantee the general passivity of the controller. Our controller is benchmarked against standard impedance control for absolute positioning accuracy across the robot workspace. Furthermore, we show its applicability to an industrial insertion task. We demonstrate absolute positioning accuracy (residual error| Ax| < 4e – 4) comparable to rigid robots while preserving compliant behavior.

Place recognition is an important technique for autonomous cars to achieve full autonomy since it can provide an initial guess to online localization algorithms. Although current methods based on images or point clouds have achieved satisfactory performance, localizing the images on a large-scale point cloud map remains a fairly unexplored problem. This cross-modal matching task is challenging due to the difficulty in extracting consistent descriptors from images and point clouds. In this paper, we propose the I2P-Rec method to solve the problem by transforming the cross-modal data into the same modality. Specifically, we leverage on the recent success of depth estimation networks to recover point clouds from images. We then project the point clouds into Bird's Eye View (BEV) images. Using the BEV image as an intermediate representation, we extract global features with a Convolutional Neural Network followed by a NetVLAD layer to perform matching. The experimental results evaluated on the KITTI dataset show that, with only a small set of training data, I2P-Rec achieves recall rates at Top-l % over 80% and 90%, when localizing monocular and stereo images on point cloud maps, respectively. We further evaluate I2P-Rec on a 1 km trajectory dataset collected by an autonomous logistics car and show that I2P- Rec can generalize well to previously unseen environments.

An important task in trajectory prediction is to model the uncertainty of agents' motions, which requires the system to propose multiple plausible future trajectories for agents based on their past movements. Recently, many approaches have been developed following an endpointconditioned deep learning framework by firstly predicting the distribution of endpoints, then sampling endpoints from it and finally completing their waypoints. However, this framework suffers a severe efficiency issue as it needs to repeatedly execute a separate decoder conditioned on multiple sampled endpoints. In this work, we propose a simple and fast endpoint conditioned fully convolutional trajectory prediction framework, called HyperTraj, by using dynamic convolutions to generate multiple trajectories, with the main benefits that (1) our prediction is conditioned on endpoint but takes almost constant time when the number of goals increases and (2) our model benefits from convolutional based predictions, such as the acceptance of various scene sizes and better modeling of agent-scene interactions. In our experiment, our model shows comparable or even better accuracy than our state-of-the-art baselines on SDD and VIRAT datasets with around 84% of acceleration and 90% model weight reduction for waypoint decoding.

3D scene graphs have recently emerged as an expressive high-level map representation that describes a 3D environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction (e.g., objects, rooms, buildings) and edges represent relations between concepts (e.g., inclusion, adjacency). This paper describes Hydra-Multi, the first multi-robot spatial perception system capable of constructing a multi-robot 3D scene graph online from sensor data collected by robots in a team. In particular, we develop a centralized system capable of constructing a joint 3D scene graph by taking incremental inputs from multiple robots, effectively finding the relative transforms between the robots' frames, and incorporating loop closure detections to correctly reconcile the scene graph nodes from different robots. We evaluate Hydra-Multi on simulated and real scenarios and show it is able to reconstruct accurate 3D scene graphs online. We also demonstrate Hydra-Multi's capability of supporting heterogeneous teams by fusing different map representations built by robots with different sensor suites.

A hybrid continuum robot design is introduced that combines a proximal tendon-actuated section with a distal telescoping section comprised of permanent-magnet spheres actuated using an external magnet. While, individually, each section can approach a point in its workspace from one or at most several orientations, the two-section combination possesses a dexterous workspace. The paper describes kinematic modeling of the hybrid design and provides a description of the dexterous workspace. We present experimental validation which shows that a simplified kinematic model produces tip position mean and maximum errors of 3% and 7% of total robot length, respectively.

Robust object pose tracking plays an important role in robot manipulation, but it is still an open issue for quickly moving targets as motion blur and low frequency detection can reduce pose estimation accuracy even for state-of-the-art RGB-D-based methods. An event-camera is a low-latency vision sensor that can act complementary to RGB-D. Specifically, its sub-millisecond temporal resolution can be exploited to correct for pose estimation inaccuracies due to low frequency RGB-D based detection. To do so, we propose a dual Kalman filter: the first filter estimates an object's velocity from the spatiotemporal patterns of “events”, the second filter fuses the tracked object velocity with a low-frequency object pose estimated from a deep neural network using RGB-D data. The full system outputs high frequency, accurate object poses also for fast moving objects. The proposed method works towards low-power robotics by replacing high-cost GPU-based optical flow used in prior work with event-cameras that inherently extract the required signal without costly processing. The proposed algorithm achieves comparable or better performance when compared to two state-of-the-art 6-DoF object pose estimation algorithms and one hybrid event/RGB-D algorithm on benchmarks with simulated and real data. We discuss the benefits and tradeoffs for using the event-camera and contribute algorithm, code, and datasets to the community. The code and datasets are available at https://github.com/event-driven-robotics/Hybrid-object-tracking-with-events-and-frames.

Fast and accurate path planning is important for ground robots to achieve safe and efficient autonomous navigation in unstructured outdoor environments. However, most existing methods exploiting either 2D or 2.5D maps struggle to balance the efficiency and safety for ground robots navigating in such challenging scenarios. In this paper, we propose a novel hybrid map representation by fusing a 2D grid and a 2.5D digital elevation map. Based on it, a novel path planning method is proposed, which considers the robot poses during traversability estimation. By doing so, our method explicitly takes safety as a planning constraint enabling robots to navigate unstructured environments smoothly. The proposed approach has been evaluated on both simulated datasets and a real robot platform. The experimental results demonstrate the efficiency and effectiveness of the proposed method. Compared to state-of-the-art baseline methods, the proposed approach consistently generates safer and easier paths for the robot in different unstructured outdoor environments. The implementation of our method is publicly available at https://github.com/nubot-nudt/T-Hybrid-planner.

This paper presents a hierarchical framework for planning and control of in-hand manipulation of a rigid object involving grasp changes using fully-actuated multifin-gered robotic hands. While the framework can be applied to the general dexterous manipulation, we focus on a more complex definition of in-hand manipulation, where at the goal pose the hand has to reach a grasp suitable for using the object as a tool. The high level planner determines the object trajectory as well as the grasp changes, i.e. adding, removing, or sliding fingers, to be executed by the low-level controller. While the grasp sequence is planned online by a learning-based policy to adapt to variations, the trajectory planner and the low-level controller for object tracking and contact force control are exclusively model-based to robustly realize the plan. By infusing the knowledge about the physics of the problem and the low-level controller into the grasp planner, it learns to successfully generate grasps similar to those generated by model-based optimization approaches, obviating the high computation cost of online running of such methods to account for variations. By performing experiments in physics simulation for realistic tool use scenarios, we show the success of our method on different tool-use tasks and dexterous hand models. Additionally, we show that this hybrid method offers more robustness to trajectory and task variations compared to a model-based method.

In order for a humanoid robot to traverse uneven terrain without falling over, the robot must control its landing position appropriately. To determine the landing position, there are two difficulties in terrain recognition and leg motion control. In terrain recognition, it is difficult to recognize and avoid terrain such as steps and obstacles that cannot be landed on in real-time. In leg motion control, it is necessary to land at appropriate positions and times to control the CoG trajectory while limiting the velocity of the swing-leg to suppress the landing impact. For solving these problems, we propose a recognition and walking control system on uneven terrain. In terrain recognition, we improved the recognition accuracy while satisfying real-time performance by using a CNN that learns the relationship between the foot and the geometric information of the surrounding terrain. In the leg motion control, landing impact was reduced by modifying the landing position under not only (1) terrain constraint and (2) robot stability constraint, but also (3) leg velocity constraint. We verified the effectiveness of the proposed system through uneven terrain walking and push recovery experiments using the actual robot.

The assistive profile of an active back support exoskeleton is strongly dependent on the manual tuning of controller gains based on previous experience and trial-and-error. Human-in-the-loop (HIL) optimization allows for automatic tuning of assistive profiles to different subjects. Most HIL methods make use of intrusive sensors that could affect out-of-the-lab exoskeleton adoption. Therefore, we propose a HIL-based assistive controller architecture using only one single IMU that can be easily embedded in any exoskeleton system. To validate our algorithm we recruited 3 subjects and asked them to perform a series of successive load liftings. Meanwhile, we analysed the back-muscles activations focusing on cumulative activation (iEMG), and median activation. We also monitored the total torque generated by the exoskeleton. With respect to an assistance-less condition, the proposed controller resulted in up to 19% reduction of the back-muscles activity. Moreover, compared to a state-of-the-art controller that produced up to 15% reduction of the back-muscles activity, the new controller also required generation of 4% less exoskeleton torque.

Contact tooling operations like sanding and polishing have been high in demand for robotics and automation, as manual operations are labour-intensive with inconsistent quality. However, automating these operations remains a challenge since they are highly dependent on prior knowledge about the geometry of the workpiece. While several methods have been developed in existing research to automate the geometry learning process and adjust the contact force, human supervision is heavily required in the calibration of workpieces and the path planning of robot motion in such methods. Furthermore, the stiffness identification of the workpiece is not considered in most of these methods. This paper presents a human-robot collaboration (HRC) framework, which is able to perform surface exploration on an unknown object combining the operator's flexibility with the control precision of the robot. The operator moves the robot along the surface of the target object, and the robot recognizes the surface geometry and surface stiffness while exerting a desired contact force through control. For this purpose, a mesh iterative learning control (MILC) is developed to learn the surface stiffness, plan the exploration path, and adjust contact force through repetitive online correction based on HRC. The proof of learning convergence and the results of the simulation and experiments performed using a 7-DOF Sawyer robot demonstrate the validity of the proposed controller.

Navigation is an essential task for social robots. However, certain rules must be followed to allow them to move without causing distraction or discomfort to people. Considering that the context surrounding robots and persons affects the expected behavior, this work defines a social area around a person that adapts to the real situation. In addition, the social context of a person is extended to identify groups of people, which the robot should take into account while navigating. With this understanding of the surrounding of the robot together with the ability to predict the trajectory of individuals as well as groups, the proposed solution not only effectively addresses collision avoidance while promoting socially acceptable behavior but also outperforms the majority of recent works in terms of accuracy. Furthermore, a dedicated policy is introduced to react to social navigation conflicts. The evaluation performed in a simulated environment shows that the computation of our proposed solution is at least 8 times faster than the best state-of-the-art approach while preserving comparable social conduct. Also, the results of realistic experiments performed using Gazebo and a real robot are reported.

When humans control or supervise remote robot manipulation, augmented reality (AR) visual cues overlaid on the remote camera video stream can effectively enhance human's remote perception of task and robot states, and comprehension of the robot autonomy's capability and intent. In this work, we conducted a user study (N=18) to investigate: (RQ1) what AR cues humans prefer when controlling the robot with various levels of autonomy, and (RQ2) whether this preference can be influenced by the way humans learn to use the interface. We provided AR visual cues of various types (e.g., motion guidance, obstacle indicator, target hint, autonomy activation and intent) to assist humans to pick and place an object around an obstacle on a counter workspace. We found that: 1) Participants prefer different types of AR cues based on the level of robot autonomy; 2) The AR cues the participants prefer to use after hands-on robot operation converged to the recommendation of experienced users, and may largely differ from their initial selection based on video instruction.

Conveying a robot's target mood is crucial to successful social interactions. The robot's expressive performance must be appropriate, persuasive, and consistent. However, this is challenging when interactions contain a mixture of scripted and improvised content, such as those generated by language models. In this paper, we take on the task of teaching robots to stay in character, that is to say, exhibit consistency in mood during interactions. We start by defining a communication strategy module that allows for the top-down specification of a target robot mood for a given task, goal, or context. We then propose a mood steering framework for enforcing robot mood consistency throughout an interaction that supports several target moods. Our framework consists of two components: 1. expressivity steering specifies the speech and behavior to be used by the robot to convey a target mood, and 2. language model steering ensures that improvised language is consistent with the robot's target mood. As a first step toward identifying effective communication strategies, we implement grumpy and cheerful strategies for a collaborative storytelling game and compare them to a neutral baseline. Evaluation in a collaborative storytelling game shows that our approach generates robot behavior that successfully conveys the robot's target mood throughout gameplay and language model steering generates story contributions that capture the target mood without quality degradation and raises important issues for communication strategy design.