IROS 2024 Papers

Exoskeleton locomotion must be robust while being adaptive to different users with and without payloads. To address these challenges, this work introduces a data-driven predictive control (DDPC) framework to synthesize walking gaits for lower-body exoskeletons, employing Hankel matrices and a state transition matrix for its data-driven model. The proposed approach leverages DDPC through a multi-layer architecture. At the top layer, DDPC serves as a planner employing Hankel matrices and a state transition matrix to generate a data-driven model that can learn and adapt to varying users and payloads. At the lower layer, our method incorporates inverse kinematics and passivity-based control to map the planned trajectory from DDPC into the full-order states of the lower-body exoskeleton. We validate the effectiveness of this approach through numerical simulations and hardware experiments conducted on the Atalante lower-body exoskeleton with different payloads. Moreover, we conducted a comparative analysis against the model predictive control (MPC) framework based on the reduced-order linear inverted pendulum (LIP) model. Through this comparison, the paper demonstrates that DDPC enables robust bipedal walking at various velocities while accounting for model uncertainties and unknown perturbations.

Landing on a vertically oscillating platform poses a significant challenge for multi-rotor unmanned aerial vehicle (UAVs) due to the time-varying ground effect (GE). In this work, we formulated a data-driven GE dynamic model that accurately describes the complex interactions between UAVs and both stationary and oscillating platforms. Integrating this model with a feedforward controller effectively compensates for GE, resulting in improved landing performance. The proposed GE model elucidates the relationship between GE and factors such as UAVs’ velocity, throttle magnitude, and the motion of the landing platform. It highlights that the GE experienced during the landing process of UAVs is not only contingent on the current state but also related to past states. The resulting GE model is parsimonious and suitable for onboard computers with limited computational power, and its accuracy has been confirmed through a series of flight experiments. To demonstrate the effectiveness of the developed UAVs landing scheme, we compared our approach with robust control and internal model control methods. Experimental results indicate that the proposed landing strategy achieves faster and smoother landings, with at least a 22% improvement in smoothness and a 25% reduction in landing time.

A novel method for analyzing the dynamics and bend geometry of a cable slab via trained neural networks is introduced. Neural networks are trained from real-time visual feedback capture via a high-speed camera during cyclic motion to track the positions of multiple markers affixed to the cable slab through image processing techniques. Experimental parameters are systematically varied to ensure a diverse range of training patterns. Consequently, two distinct data-driven neural network models are developed: a coupled model and a decoupled model. These models accurately predict the two-dimensional positions of the markers, even during non-cyclic motion profiles. Subsequently, the marker positions are utilized as waypoints to generate a cubic spline curve with time-varying coefficients, approximating the spatiotemporal solution of the cable slab dynamics. Notably, this spline can be segmented into smaller sections tailored to specific research objectives. Experimental results validate the effectiveness of the proposed methodology.

Highly dynamic maneuvers pose a challenge to conventional state estimators of quadrotors in rapidly tracking the pose. This paper proposes a data-driven Koopman operator-based error-state Kalman filter (K-ESKF) to enhance pose estimation in agile flight. Our method uses the Koopman operator theory to transform the full-state nonlinear quadrotor dynamics into a lifted bilinear control system driven by accelerations and angular rates. A deep neural network (DNN) is used to represent the Koopman observable functions. Our proposed K-ESKF extends the propagation step of a standard error-state Kalman filter (ESKF) using the lifted bilinear control system. An open-source quadrotor dataset, NeuroBEM, is used for training and evaluating the DNN and for testing the K-ESKF. The learned Koopman bilinear system demonstrates a 60% less attitude errors compared to the first-order Euler method in terms of model accuracy. Using real trajectories from the dataset, our proposed K-ESKF can estimate the pose as accurately as the ESKF during normal flight. More importantly, our proposed approach outperforms the ESKF by achieving about 50% less attitude and velocity estimation errors in a highly agile flight. During drastic attitude and velocity changes, the K-ESKF can still estimate the pose while the ESKF loses tracking.

We consider imitation learning with access only to expert demonstrations, whose real-world application is often limited by covariate shift due to compounding errors during execution. We investigate the effectiveness of the Continuity-based Corrective Labels for Imitation Learning (CCIL) framework in mitigating this issue for real-world fine manipulation tasks. CCIL generates corrective labels by learning a locally continuous dynamics model from demonstrations to guide the agent back toward expert states. Through extensive experiments on insertion and fine grasping tasks, we provide the first empirical validation that CCIL can significantly improve imitation learning performance despite discontinuities present in contact-rich manipulation. We find that: (1) real-world manipulation exhibits sufficient local smoothness to apply CCIL, (2) generated corrective labels are most beneficial in low-data regimes, and (3) label filtering based on estimated dynamics model error enables performance gains. To effectively apply CCIL to robotic domains, we offer a practical instantiation of the framework and insights into design choices and hyperparameter selection. Our work demonstrates CCIL’s practicality for alleviating compounding errors in imitation learning on physical robots.

Humans have the remarkable ability to construct consistent mental models of an environment, even under limited or varying levels of illumination. We wish to endow robots with this same capability. In this paper, we tackle the challenge of constructing a photorealistic scene representation under poorly illuminated conditions and with a moving light source. We approach the task of modeling illumination as a learning problem, and utilize the developed illumination model to aid in scene reconstruction. We introduce an innovative framework that uses a data-driven approach, Neural Light Simulators (NeLiS), to model and calibrate the camera-light system. Furthermore, we present DarkGS, a method that applies NeLiS to create a relightable 3D Gaussian scene model capable of real-time, photorealistic rendering from novel viewpoints. We show the applicability and robustness of our proposed simulator and system in a variety of real-world environments. Code released at https://tyz1030.github.io/proj/darkgs.html

Recognizing daily human activities is essential for domestic robots to assist humans effectively in indoor environments. These activities typically involve sequences of interactions between humans and objects across different locations and times within a household. Identifying these events and understanding their temporal and spatial relationships is crucial for accurately modeling human behavior patterns. However, most current methods and datasets for human activity recognition focus on identifying singular events at specific moments and locations, neglecting the complexity of activities that span multiple times and places. To address this gap, we collected data on human activity patterns over a single day through crowdsourcing. Based on this, we introduce a novel synthetic video question-answering dataset. Our proposed dataset includes videos of daily activities accompanied by question-answer pairs that require models to reason about sequences of activities in both time and space. We evaluated state-of-the-art methods against our dataset, highlighting their limitations in handling the intricate spatio-temporal dynamics of human activity sequences. To improve upon these methods, we propose a two-stage model. The proposed model initially decodes the detailed content of individual videos using a transformer-based approach, then employs LLMs for advanced spatio-temporal reasoning across multiple videos. We hope our research provides valuable benchmarks and insights, paving the way for advancements in the recognition of daily human activity patterns.

Domain adaptation is an inspiring solution to the misalignment issue of day/night image features for nighttime UAV tracking. However, the one-step adaptation paradigm is inadequate in addressing the prevalent difficulties posed by low-resolution (LR) objects when viewed from the UAVs at night, owing to the blurry edge contour and limited detail information. Moreover, these approaches struggle to perceive LR objects disturbed by nighttime noise. To address these challenges, this work proposes a novel progressive alignment paradigm, named domain-aware diffusion model (DaDiff), aligning nighttime LR object features to the daytime by virtue of progressive and stable generations. The proposed DaDiff includes an alignment encoder to enhance the detail information of nighttime LR objects, a tracking-oriented layer designed to achieve close collaboration with tracking tasks, and a successive distribution discriminator presented to distinguish different feature distributions at each diffusion timestep successively. Furthermore, an elaborate nighttime UAV tracking benchmark is constructed for LR objects, namely NUT-LR, consisting of 100 annotated sequences. Exhaustive experiments have demonstrated the robustness and feature alignment ability of the proposed DaDiff. The source code and video demo are available at https://github.com/vision4robotics/DaDiff.

Monocular Bird’s Eye View (BEV) semantic segmentation is critical for autonomous driving for its inherent advantages in spatial representation and downstream tasks. However, it is challenging to simultaneously learn view transformation and pixel-wise classification. Previous works suffer from non-flat region distortion, distant depth ambiguity, and visual occlusion. To address these aforementioned concerns, we propose dual-branch view transformation (DVT), a novel framework for monocular BEV semantic segmentation. Our method consists of: (i) A dual-branch view transformation to decouple features into flat region and non-flat region and process them independently. (ii) A depth-aware weighting method to make the model pay more attention to the distant depth. (iii) An auxiliary task to introduce more inductive biases to alleviate the inaccuracy caused by visual occlusion. Furthermore, we design a class-aware weighting method to address the class and size imbalance of datasets. Experimental results on nuScenes and KITTI-360 datasets demonstrate that DVT outperforms previous state-of-the-art (SOTA). Our codes are available at https://github.com/MrPicklesGG/DVT.

We present a novel end-to-end diffusion-based trajectory generation method, DTG, for mapless global navigation in challenging outdoor scenarios with occlusions and unstructured off-road features like grass, buildings, bushes, etc. Given a distant goal, our approach computes a trajectory that satisfies the following goals: (1) minimize the travel distance to the goal; (2) maximize the traversability by choosing paths that do not lie in undesirable areas. Specifically, we present a novel Conditional RNN(CRNN) for diffusion models to efficiently generate trajectories. Furthermore, we propose an adaptive training method that ensures that the diffusion model generates more traversable trajectories. We evaluate our methods in various outdoor scenes and compare the performance with other global navigation algorithms on a Husky robot. In practice, we observe at least a 15% improvement in traveling distance and around a 7% improvement in traversability. Video and Code: https://github.com/jingGM/DTG.git.

In recent years, 3D surround view systems have attracted a lot of attention in the field of advanced driver assistance systems (ADAS). However, the foundational assumption of unchanging camera poses in traditional 3D surround view systems, which is designed for single-unit vehicles, results in a failure to manage the non-rigid connections characteristic of tractor-trailer vehicles. Moreover, tractor-trailer vehicles have the feature of long bodies and large wheelbases, leading to severe distortions and abrupt changes in the rendering results of previous 3D texture mapping models. In this paper, we propose DSVT, a dynamic 3D surround view system for tractor-trailer vehicles, designed to address the aforementioned issues. Specifically, we develop a dynamic surround image stitching algorithm based on relative pose estimation, which estimates the relative poses between cameras and stitches all images together to generate a 2D panoramic image. Subsequently, a novel 3D drop model is proposed, mapping the 2D panoramic image onto the 3D model for panoramic viewing. Our system can run in real time on Nvidia AGX Orin. Experimental results in real tractor-trailer scenes show that our system can achieve more accurate and natural visual effects.

This paper introduces a 3D point cloud sequence learning model based on inconsistent spatio-temporal propagation for LiDAR odometry, termed DSLO. It consists of a pyramid structure with a spatial information reuse strategy, a sequential pose initialization module, a gated hierarchical pose refinement module, and a temporal feature propagation module. First, spatial features are encoded using a point feature pyramid, with features reused in successive pose estimations to reduce computational overhead. Second, a sequential pose initialization method is introduced, leveraging the high-frequency sampling characteristic of LiDAR to initialize the LiDAR pose. Then, a gated hierarchical pose refinement mechanism refines poses from coarse to fine by selectively retaining or discarding motion information from different layers based on gate estimations. Finally, temporal feature propagation is proposed to incorporate the historical motion information from point cloud sequences, and address the spatial inconsistency issue when transmitting motion information embedded in point clouds between frames. Experimental results on the KITTI odometry dataset and Argoverse dataset demonstrate that DSLO outperforms state-of-the-art methods, achieving at least a 15.67% improvement on RTE and a 12.64% improvement on RRE, while also achieving a 34.69% reduction in runtime compared to baseline methods. Our implementation will be available at https://github.com/IRMVLab/DSLO.

Unmanned Aerial Vehicles (UAVs) play a crucial role in meteorological research, particularly in environmental wind field measurements. However, several challenges exist in current wind measurement methods using UAVs that need to be addressed. Firstly, the accuracy of measurement is low, and the measurement range is limited. Secondly, the algorithms employed lack robustness and adaptability across different UAV platforms. Thirdly, there are limited approaches available for wind estimation during dynamic flight. Finally, while horizontal plane measurements are feasible, vertical direction estimation is often missing. To tackle these challenges, we present and implement a comprehensive wind estimation algorithm. Our algorithm offers several key features, including the capability to estimate the 3-D wind vector, enabling wind estimation even during dynamic flight of the UAV. Furthermore, our algorithm exhibits adaptability across various UAV platforms. Experimental results in the wind tunnel validate the effectiveness of our algorithm, showcasing improvements such as wind speed accuracy of 0.11 m/s and wind direction errors of less than 2.8°. Additionally, our approach extends the measurement range to 10 m/s.Wind Estimation, Disturbance Observer, Aerial Robotics, Field Robots

In recent years, coordinate-based neural implicit representations have shown promising results for the task of Simultaneous Localization and Mapping (SLAM). While achieving impressive performance on small synthetic scenes, these methods often suffer from losing details, especially for complex real-world scenes. In this work, we introduce DNS SLAM, a novel neural RGB-D semantic SLAM approach featuring a hybrid representation. Relying only on 2D semantic priors, we propose the first semantic neural SLAM method that trains class-wise scene representations while providing stable camera tracking at the same time. Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details and to output color, occupancy, and semantic class information, enabling many downstream applications. To further enable fast tracking, we introduce a lightweight coarse scene representation which is trained in a self-supervised manner in latent space. Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking while maintaining a commendable operational speed on off-the-shelf hardware. Further, our method outputs class-wise decomposed reconstructions with better texture, capturing appearance and geometric details.

Teaching robots new skills quickly and conveniently is crucial for the broader adoption of robotic systems. In this work, we address the problem of one-shot imitation from a single human demonstration, given by an RGB-D video recording. We propose a two-stage process. In the first stage we extract the demonstration trajectory offline. This entails segmenting manipulated objects and determining their relative motion in relation to secondary objects such as containers. In the online trajectory generation stage, we first re-detect all objects, then warp the demonstration trajectory to the current scene and execute it on the robot. To complete these steps, our method leverages several ancillary models, including those for segmentation, relative object pose estimation, and grasp prediction. We systematically evaluate different combinations of correspondence and re-detection methods to validate our design decision across a diverse range of tasks. Specifically, we collect and quantitatively test on demonstrations of ten different tasks including pick-and-place tasks as well as articulated object manipulation. Finally, we perform extensive evaluations on a real robot system to demonstrate the effectiveness and utility of our approach in real-world scenarios. We make the code publicly available at http://ditto.cs.uni-freiburg.de.

Wheeled bipedal robots offer the advantages of both wheeled and legged robots, combining the ability to traverse a wide range of terrains and environments with high efficiency. However, the conventional approach in existing wheeled bipedal robots involves motor-driven joints with high-ratio gearboxes. While this approach provides specific benefits, it also presents several challenges, including increased mechanical complexity, efficiency losses, noise, vibrations, and higher maintenance and lubrication requirements.Addressing the aforementioned concerns, we developed a direct-drive wheeled bipedal robot called DIABLO, which eliminates the use of gearboxes entirely. Our robotic system is simplified as a second-order inverted pendulum, and we have designed an LQR-based balance controller to ensure stability. Additionally, we implemented comprehensive motion controller, including yaw, split-angle, height, and roll controllers. Through experiments in both simulations and real-world prototypes, we have demonstrated that our platform achieves satisfactory performance.

Maps provide robots with crucial environmental knowledge, thereby enabling them to perform interactive tasks effectively. Easily accessing accurate abstract-to-detailed geometric and semantic concepts from maps is crucial for robots to make informed and efficient decisions. To comprehensively model the environment and effectively manage the map data structure, we propose DHP-Mapping, a dense mapping system that utilizes multiple Truncated Signed Distance Field (TSDF) submaps and panoptic labels to hierarchically model the environment. The output map is able to maintain both voxel- and submap-level metric and semantic information. Two modules are presented to enhance the mapping efficiency and label consistency: (1) an inter-submaps label fusion strategy to eliminate duplicate points across submaps and (2) a conditional random field (CRF) based approach to enhance panoptic labels. We conducted experiments with two public datasets including indoor and outdoor scenarios. Our system performs comparably to state-of-the-art (SOTA) methods across geometry and label accuracy evaluation metrics. The experiment results highlight the effectiveness and scalability of our system, as it is capable of constructing precise geometry and maintaining consistent panoptic labels. Our code is publicly available at https://github.com/hutslib/DHP-Mapping.

Endoscopic Submucosal Dissection (ESD) is a minimally invasive procedure designed to remove precancerous and cancerous lesions from the gastrointestinal (GI) tract. Given the GI tract’s tortuous and narrow shape, along with the need for varied movements during dissection, this requires highly flexible and compact instruments, making flexible continuum robots suitable candidates. In this paper, we propose a novel two-segment continuum robot system named DESectBot, featuring a diameter of 5.5 mm and a total length of the active bending module of 48 mm, while the robot’s total length exceeds 1 m. We designed a novel joint combination structure called the spatial cross-curved disk skeleton for the robot, which addresses the mechanical coupling problem between flexible robot actuators. The DESectBot boasts six degrees of freedom, and its kinematic modeling has been derived and utilized in the closed-loop control of the DESectBot. The validation of the DESectBot was conducted through a two-stage test: first, the decoupling performance of the DESectBot was validated. The results show that when one active bending segment bends, the other segment remains almost uninfluenced, with a maximum variation of 1.15 degrees, demonstrating the robot’s effective decoupling capability. Secondly, the accuracy of DESectBot was validated through trajectory-following experiments. The results reveal that the average tracking error for both trajectories is less than 2 mm, and the maximum tracking error is below 2.5 mm. Taking marking, one of the ESD procedures with a 5mm tolerance, as an example, the DESectBot has the potential to be utilized for ESD procedure.

This paper proposes a task planning framework for collaborative Human-Robot scenarios, specifically focused on assembling complex systems such as furniture. The human is characterized as an uncontrollable agent, implying for example that the agent is not bound by a pre-established sequence of actions and instead acts according to its own preferences. Meanwhile, the task planner computes reactively the optimal actions for the collaborative robot to efficiently complete the entire assembly task in the least time possible.We formalize the problem as a Discrete Event Markov Decision Problem (DE-MDP), a comprehensive framework that incorporates a variety of asynchronous behaviors, human change of mind, and failure recovery as stochastic events. Although the problem could theoretically be addressed by constructing a graph of all possible actions, such an approach would be constrained by computational limitations. The proposed formulation offers an alternative solution utilizing Reinforcement Learning to derive an optimal policy for the robot. Experiments were conducted both in simulation and on a real system with human subjects assembling a chair in collaboration with a 7-DoF manipulator.