ICRA 2024 论文列表

In order to enhance driving convenience and safety, automotive Augmented Reality displays, e.g., head-up displays, have garnered attention and are gradually being deployed. However, when vehicles encounter uneven roads, vertical vibrations lead to mismatches between external physical objects and augmented reality overlay images, adversely affecting the AR display’s visibility. Resolving the problem is quite challenging because the optical system operates on a nanometer scale and is highly sensitive due to its multifunctional nature involving reflection and refraction through an intermediate medium. This paper aims to address the newly emerging problem of vertical mismatches in automotive AR displays. To tackle this issue, we begin by defining the problem and then examine the effectiveness of traditional control methods, on-policy and off-policy reinforcement learning as potential solutions. Finally, we validate our approach through experiments, demonstrating a significant reduction in vertical mismatches and an improvement in the overall visibility of automotive AR displays. Our findings provide valuable insights for enhancing driving convenience and safety in real-world conditions.

Understanding actions of other agents increases the efficiency of autonomous mobile robots (AMRs) since they encompass intention and indicate future movements. We propose a new method that allows us to infer vehicle actions using a shallow image-based classification model. The actions are classified via bird’s-eye view scene crops, where we project the detections of a 3D object detection model onto a context map. We learn map context information and aggregate temporal sequence information without requiring object tracking. This results in a highly efficient classification model that can easily be deployed on embedded AMR hardware. To evaluate our approach, we create new large-scale synthetic datasets showing warehouse traffic based on real vehicle models and geometry.

This paper presents a 2D skeleton-based action segmentation method with applications in fine-grained human activity recognition. In contrast with state-of-the-art methods which directly take sequences of 3D skeleton coordinates as inputs and apply Graph Convolutional Networks (GCNs) for spatiotemporal feature learning, our main idea is to use sequences of 2D skeleton heatmaps as inputs and employ Temporal Convolutional Networks (TCNs) to extract spatiotemporal features. Despite lacking 3D information, our approach yields comparable/superior performances and better robustness against missing keypoints than previous methods on action segmentation datasets. Moreover, we improve the performances further by using both 2D skeleton heatmaps and RGB videos as inputs. To our best knowledge, this is the first work to utilize 2D skeleton heatmap inputs and the first work to explore 2D skeleton+RGB fusion for action segmentation.

Collaborative perception leverages rich visual observations from multiple robots to extend a single robot’s perception ability beyond its field of view. Many prior works receive messages broadcast from all collaborators, leading to a scalability challenge when dealing with a large number of robots and sensors. In this work, we aim to address scalable camera-based collaborative perception with a Transformer-based architecture. Our key idea is to enable a single robot to intelligently discern the relevance of the collaborators and their associated cameras according to a learned spatial prior. This proactive understanding of the visual features’ relevance does not require the transmission of the features themselves, enhancing both communication and computation efficiency. Specifically, we present ActFormer, a Transformer that learns bird’s eye view (BEV) representations by using predefined BEV queries to interact with multi-robot multi-camera inputs. Each BEV query can actively select relevant cameras for information aggregation based on pose information, instead of interacting with all cameras indiscriminately. Experiments on the V2X-Sim dataset demonstrate that ActFormer improves the detection performance from 29.89% to 45.15% in terms of AP@0.7 with about 50% fewer queries, showcasing the effectiveness of ActFormer in multi-agent collaborative 3D object detection.

Micromanipulation techniques that can achieve controlled fine operations at the micro scale play an important role in biomedical fields including embryo engineering, gene engineering, drug screening, and cell analysis. However, micromanipulation of biological micro-objects, such as cells and micro tissues, suffers from mechanical damage and low efficiency. Several techniques have been introduced to manipulate cells more easily, but most of them are restricted by expensive devices, limited work area, and potential damage to cellular structure. Here we develop a hydrodynamic manipulation method to rotate and transport mouse oocytes, which utilizes acoustic waves and micropipette to generate acoustic radiation force and excite microstreaming. This method can accomplish rotational and translational operations precisely and controllably. We tested the process of trapping, rotation, and transportation of the mouse oocytes, and measured rotational and translational speed with a range of applied voltage. The method was able to shorten the cost time of delivery and posture adjustment before oocyte injection. Our study provides an easy-to-use technique for oocyte manipulation without contact, and it has the potential to be universally applied in many cellular studies.

This paper presents a novel acoustic soft tactile (AST) skin technology operating with sound waves. In this innovative approach, the sound waves generated by a speaker travel in channels embedded in a soft membrane and get modulated due to a deformation of the channel when pressed by an external force and received by a microphone at the end of the channel. The sensor leverages regression and classification methods for estimating the normal force and its contact location. Our sensor can be affixed to any robot part, e.g., end effectors or arm. We tested several regression and classifier methods to learn the relation between sound wave modulation, the applied force, and its location, respectively and picked the best-performing models for force and location predictions. The best skin configurations yield more than 93% of the force estimation within ±1.5 N tolerances for a range of 0-30+1 N and contact locations with over 96% accuracy. We also demonstrated the performance of AST Skin technology for a real-time gripping force control application.

Recently, the importance of mechanical transparency in human-assistive robots has grown. Traditionally, its primary goal was minimizing interaction forces during assistance. However, under this conventional definition, mechanical transparency was not considered when an interaction force was required during assistance. This research focuses on achieving mechanical transparency within the context of shoulder motion in upper extremity exoskeletons for rehabilitation. Our primary goal is maintaining interaction forces at target values, even with motion disturbances. To this end, we developed a shoulder actuation testbed for exoskeletons, incorporating a fusion hybrid linear actuator distinguished by high back-drivability, robust torque generation capability, and safety features. To attain mechanical transparency, we created a model for calculating the required joint torque, accounting for gravitational dynamics, and subsequently determined the necessary actuator output. The system characteristics were evaluated based on the joint torque generated by the actuator. The actuator utilized pneumatic pressure to generate force and compensated for kinetic friction using electromagnetic forces. The results showed that the compensation by the electromagnetic force reduced the root mean square error of the torque to less than 60% in relation to pneumatic pressure alone. This demonstrated the ability to generate consistent torque with high robustness to motion disturbances.

Cloth manipulation is a category of deformable object manipulation of great interest to the robotics community, from applications of automated laundry-folding and home organizing to textiles and flexible manufacturing. Despite the desire for automated cloth manipulation, the thin-shell dynamics and under-actuation nature of cloth present significant challenges for robots to effectively interact with them. Many recent works omit explicit modeling in favor of learning-based methods that may yield control policies directly. However, these methods require large training sets that must be collected and curated. In this regard, we create a framework for differentiable modeling of cloth dynamics leveraging an Extended Position-based Dynamics (XPBD) algorithm. Together with the desired control objective, physics-aware regularization terms are designed for better results, including trajectory smoothness and elastic potential energy. In addition, safety constraints, such as avoiding obstacles, can be specified using signed distance functions (SDFs). We formulate the cloth manipulation task with safety constraints as a constrained optimization problem, which can be effectively solved by mainstream gradient-based optimizers thanks to the end-to-end differentiability of our framework. Finally, we assess the framework with various safety thresholds and demonstrate the feasibility of result trajectories on a surgical robot. The effects of the regularization terms are analyzed in an additional ablation study.

Unmanned vehicles usually rely on Global Positioning System (GPS) and Light Detection and Ranging (LiDAR) sensors to achieve high-precision localization results for navigation purpose. However, this combination with their associated costs and infrastructure demands, poses challenges for widespread adoption in mass-market applications. In this paper, we aim to use only a monocular camera to achieve comparable onboard localization performance by tracking deep-learning visual features on a LiDAR-enhanced visual prior map. Experiments show that the proposed algorithm can provide centimeter-level global positioning results with scale, which is effortlessly integrated and favorable for low-cost robot system deployment in real-world applications.

In robotics and biomechanics, accurately determining joint parameters and computing the corresponding forward and inverse kinematics are critical yet often challenging tasks, especially when dealing with highly individualized and partly unknown systems. This paper unveils a cutting-edge kinematic optimizer, underpinned by an autoencoder-based architecture, to address these challenges. Utilizing a neural network, our approach simulates inverse kinematics, converting measurement data into joint-specific parameters during encoding, enabling a stable optimization process. These parameters are subsequently processed through a predefined, differentiable forward kinematics model, resulting in a decoded representation of the original data. Beyond offering a comprehensive solution to kinematics challenges, our method also unveils previously unidentified joint parameters. Real experimental data from knee and hand joints validate the optimizer’s efficacy. Additionally, our optimizer is multifunctional: it streamlines the modeling and automation of kinematics and enables a nuanced evaluation of diverse modeling techniques. By assessing the differences in reconstruction losses, we illuminate the merits of each approach. Collectively, this preliminary study signifies advancements in kinematic optimization, with potential applications spanning both biomechanics and robotics.

Many applications require the deployment of legged-robot teams to effectively and efficiently carry out missions. The use of multiple robots allows tasks to be executed concurrently, expediting mission completion. It also enhances resilience by enabling task transfer in case of a robot failure. This paper presents a formulation based on Mixed Integer Linear Programming (MILP) for allocating tasks to robots by taking into account travel time and ensuring efficient execution of collaborative tasks. We extended the MILP formulation to account for complexities with legged robot teams. Our results demonstrate that this approach leads to improved performance in terms of the makespan of the mission. We demonstrate the usefulness of this approach using a case study involving the disinfection of a building consisting of multiple rooms.

Industrial robots are capable of moving at high speed. Each time they come into contact with their environment, e.g. to pick up an object, they decelerate to a near standstill. A solution involving a compliant pneumatic gripper and adapted trajectory plan is presented to initiate contact at a higher speed while remaining within hardware limits. By adding overload clutches in either the robot arm or gripper, tolerance to errors is provided. The key parameters such as gripper compliance and maximum allowed initial impact velocity are identified. Results show that by properly optimizing these parameters, robot picking of rigid objects can be accelerated. The complete high-speed picking solution is experimentally verified. A time reduction of 16% was obtained when making contact at 0.65 m/s.

Long-horizon task planning is important for robot autonomy, especially as a subroutine for frameworks such as Integrated Task and Motion Planning. However, task planning is computationally challenging and struggles to scale to realistic problem settings. We propose to accelerate task planning over an agent’s lifetime by integrating abstract strategies: a generalizable planning experience encoding introduced in earlier work. In this work, we contribute a practical approach to planning with strategies by introducing a novel formalism of planning in a strategy-augmented domain. We also introduce and formulate the notion of a strategy’s affordance, which indicates its predicted benefit to the solution, and use it to guide the planning and strategy grounding processes. Together, our observations yield an affordance-directed, lazy-search planning algorithm, which can seamlessly compose strategies and actions to solve long-horizon planning problems. We evaluate our planner in an object rearrangement domain, where we demonstrate performance benefits relative to a state-of-the-art task planner.

Tactile exploration plays a crucial role in understanding object structures for fundamental robotics tasks such as grasping and manipulation. However, efficiently exploring such objects using tactile sensors is challenging, primarily due to the large-scale unknown environments and limited sensing coverage of these sensors. To this end, we present AcTExplore, an active tactile exploration method driven by reinforcement learning for object reconstruction at scales that automatically explores the object surfaces in a limited number of steps. Through sufficient exploration, our algorithm incrementally collects tactile data and reconstructs 3D shapes of the objects as well, which can serve as a representation for higher-level downstream tasks. Our method achieves an average of 95.97% IoU coverage on unseen YCB objects while just being trained on primitive shapes.

Combining LiDAR and camera data has shown potential in enhancing short-distance object detection in autonomous driving systems. Yet, the fusion encounters difficulties with extended distance detection due to the contrast between LiDAR’s sparse data and the dense resolution of cameras. Besides, discrepancies in the two data representations further complicate fusion methods. We introduce AYDIV, a novel framework integrating a tri-phase alignment process specifically designed to enhance long-distance detection even amidst data discrepancies. AYDIV consists of the Global Contextual Fusion Alignment Transformer (GCFAT), which improves the extraction of camera features and provides a deeper understanding of large-scale patterns; the Sparse Fused Feature Attention (SFFA), which fine-tunes the fusion of LiDAR and camera details; and the Volumetric Grid Attention (VGA) for a comprehensive spatial data fusion. AYDIV’s performance on the Waymo Open Dataset (WOD) with an improvement of 1.24% in mAPH value(L2 difficulty) and the Argoverse2 Dataset with a performance improvement of 7.40% in AP value demonstrates its efficacy in comparison to other existing fusion-based methods. Our code is publicly available at https://github.com/sanjay-810/AYDIV2

Point cloud prediction is an important yet challenging task in the field of autonomous driving. The goal is to predict future point cloud sequences that maintain object structures while accurately representing their temporal motion. These predicted point clouds help in other subsequent tasks like object trajectory estimation for collision avoidance or estimating locations with the least odometry drift. In this work, we present ATPPNet, a novel architecture that predicts future point cloud sequences given a sequence of previous time step point clouds obtained with LiDAR sensor. ATPPNet leverages Conv-LSTM along with channel-wise and spatial attention dually complemented by a 3D-CNN branch for extracting an enhanced spatio-temporal context to recover high quality fidel predictions of future point clouds. We conduct extensive experiments on publicly available datasets and report impressive performance outperforming the existing methods. We also conduct a thorough ablative study of the proposed architecture and provide an application study that highlights the potential of our model for tasks like odometry estimation.

This paper proposes an informative trajectory planning approach, namely, adaptive particle filter tree with sigma point-based mutual information reward approximation (ASPIRe), for mobile target search and tracking (SAT) in cluttered environments with limited sensing field of view. We develop a novel sigma point-based approximation to accurately estimate mutual information (MI) for general, non-Gaussian distributions utilizing particle representation of the belief state, while simultaneously maintaining high computational efficiency. Building upon the MI approximation, we develop the Adaptive Particle Filter Tree (APFT) approach with MI as the reward, which features belief state tree nodes for informative trajectory planning in continuous state and measurement spaces. An adaptive criterion is proposed in APFT to adjust the planning horizon based on the expected information gain. Simulations and physical experiments demonstrate that ASPIRe achieves real-time computation and outperforms benchmark methods in terms of both search efficiency and estimation accuracy.

Low-light enhancement and deblurring is vital for high-level vision-related nighttime tasks. Most existing cascade and joint enhancement methods may provide undesirable results, suffering from severe artifacts, deteriorating blur, and unclear details. In this paper, we propose a novel ambiguity-aware network (ASP-LED) with structural priors, including high-frequency and edge, to enable effective image representation learning for joint low-light enhancement and deblurring. Specifically, we employ a Transformer backbone to explore the global clues of the image. To compensate for the inadequate local detail optimization, we propose a multi-patch perception pyramid block that models the correlation between different size patches and ambiguity, and identifies non-uniform deblurring spatial features, facilitating the reconstruction of potential high-frequency and edge information. Furthermore, a prior-guided reconstruction block based on the parallel attention mechanism is present to adaptively correct global image with statistical features, which helps guide the model to refine sharp texture and structure. Extensive experiments performed on simulated and real-world datasets demonstrate the efficacy of our proposed method in restoring low-light blurry images with increased visual perception compared to state-of-the-art methods.

In this paper, we tackle the problem of grasping transparent and specular objects. This issue holds importance, yet it remains unsolved within the field of robotics due to failure of recover their accurate geometry by depth cameras. For the first time, we propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. ASGrasp utilizes a two-layer learning-based stereo network for the purpose of transparent object reconstruction, enabling material-agnostic object grasping in cluttered environments. In contrast to existing RGB-D based grasp detection methods, which heavily depend on depth restoration networks and the quality of depth maps generated by depth cameras, our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction. We create an extensive synthetic dataset through domain randomization, which is based on GraspNet-1Billion. Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping in both simulation and the real via seamless sim-to-real transfer. Our method significantly outperforms SOTA networks and even surpasses the performance upper bound set by perfect visible point cloud inputs. Project page: https://pku-epic.github.io/ASGrasp