ICRA 2023 论文列表

We present a learning algorithm, DifFAR, for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras that contain a human actor along with background motion. Typically, the human actors occupy less than one-tenth of the spatial resolution. DifFAR simultaneously harnesses the benefits of frequency domain representations, a classical analysis tool in signal processing, and data driven neural networks. We build a differentiable static-dynamic frequency mask prior to model the salient static and dynamic pixels in the video, crucial for the underlying task of action recognition. We use this differentiable mask prior to enable the neural network to intrinsically learn disentangled feature representations via an identity loss function. Our formulation empowers the network to inherently compute disentangled salient features within its layers. Further, we propose a cost-function encapsulating temporal relevance and spatial content to sample the most important frame within uniformly spaced video segments. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset and demonstrate relative improvements of 5.72% - 13.00% over the state-of-the-art and 14.28% - 38.05% over the corresponding baseline model.

Dextrous in-hand manipulation with a multi-fingered robotic hand is a challenging task, esp. when performed with the hand oriented upside down, demanding permanent force-closure, and when no external sensors are used. For the task of reorienting an object to a given goal orientation (vs. infinitely spinning it around an axis), the lack of external sensors is an additional fundamental challenge as the state of the object has to be estimated all the time, e.g., to detect when the goal is reached. In this paper, we show that the task of reorienting a cube to any of the 24 possible goal orientations in a π/2-raster using the torque-controlled DLR-Hand II is possible. The task is learned in simulation using a modular deep reinforcement learning architecture: the actual policy has only a small observation time window of 0.5 s but gets the cube state as an explicit input which is estimated via a deep differentiable particle filter trained on data generated by running the policy. In simulation, we reach a success rate of 92% while applying significant domain randomization. Via zero-shot Sim2Real-transfer on the real robotic system, all 24 goal orientations can be reached with a high success rate. (Web: dlr-alr.github.io/dlr-tactile-manipulation)

Complex and contact-rich robotic manipulation tasks, particularly those that involve multi-fingered hands and underactuated object manipulation, present a significant challenge to any control method. Methods based on reinforcement learning offer an appealing choice for such settings, as they can enable robots to learn to delicately balance contact forces and dexterously reposition objects without strong modeling assumptions. However, running reinforcement learning on real-world dexterous manipulation systems often requires significant manual engineering. This negates the benefits of autonomous data collection and ease of use that reinforcement learning should in principle provide. In this paper, we describe a system for vision-based dexterous manipulation that provides a “programming-free” approach for users to define new tasks and enable robots with complex multi-fingered hands to learn to perform them through interaction. The core principle under-lying our system is that, in a vision-based setting, users should be able to provide high-level intermediate supervision that circumvents challenges in teleoperation or kinesthetic teaching which allows a robot to not only learn a task efficiently but also to autonomously practice. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples, a reinforcement learning procedure that learns the task autonomously without interventions, and experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world, without simulation, manual modeling, or reward engineering.

Optimizing behaviors for dexterous manipulation has been a longstanding challenge in robotics, with a variety of methods from model-based control to model-free reinforcement learning having been previously explored in literature. Such prior work often require extensive trial-and-error training along with task-specific tuning of reward functions, which makes applying dexterous manipulation for general purpose problems quite impractical. A sample-efficient and practical alternate to trial-and-error learning is imitation learning. However, collecting and learning from demonstrations in dexterous manipulation is quite challenging due to the high-dimensional action-space involved with multi-finger control. In this work, we propose ‘Dexterous Imitation Made Easy’ (DIME) a new imitation learning framework for dexterous manipulation. DIME only requires a single RGB camera that observes a human operator to teleoperate a robotic hand. Once demonstrations are collected, DIME employs state-of-the-art imitation learning methods to train dexterous manipulation policies. On real robot benchmarks we demonstrate that DIME can be used to solve complex, in-hand manipulation tasks such as ‘flipping’, ‘spinning’, and ‘rotating’ objects with just 30 demonstrations and no additional robot training. Our code, pre-collected demonstrations, and robot videos are publicly available at: https://nyu-robot-learning.github.io/dime.

Robotic dexterous grasping is the first step to enable human-like dexterous object manipulation and thus a crucial robotic technology. However, dexterous grasping is much more under-explored than object grasping with parallel grippers, partially due to the lack of a large-scale dataset. In this work, we present a large-scale robotic dexterous grasp dataset, DexGraspNet, generated by our proposed highly efficient synthesis method that can be generally applied to any dexterous hand. Our method leverages a deeply accelerated differentiable force closure estimator and thus can efficiently and robustly synthesize stable and diverse grasps on a large scale. We choose ShadowHand and generate 1.32 million grasps for 5355 objects, covering more than 133 object categories and containing more than 200 diverse grasps for each object instance, with all grasps having been validated by the Isaac Gym simulator. Compared to the previous dataset from Liu et al. generated by GraspIt!, our dataset has not only more objects and grasps, but also higher diversity and quality. Via performing cross-dataset experiments, we show that training several algorithms of dexterous grasp synthesis on our dataset significantly outperforms training on the previous one. To access our data and code, including code for human and Allegro grasp synthesis, please visit our project page: https://pku-epic.github.io/DexGraspNet/.

In this study, we develop a hydraulically-driven soft robotic hand for handling heavy vegetables in a vegetable factory and report its experimental validations. The working population in agriculture is decreasing worldwide, creating a lot of demands for the robotic automation in harvest and trans-portation of agricultural produces. In particular, a vegetable factory deals with large and heavy vegetables, e.g., cabbages, with 2–3 kg weight and 20–30 cm diameter. A soft robot hand is suitable for handling a food or vegetable; however, most of existing soft robot hands cannot generate necessary output because they are usually actuated by the air-pressure. Therefore, we employ the hydraulic actuation for our soft hand to generate 1 or 2 MPa pressure. Using the developed soft hand, we report experimental validations including basic control performance evaluation and grasping experiments assuming a vegetable factory environment.

Robot-assisted percutaneous needle insertion is expected to significantly increase targeting accuracy in minimally invasive operations. For this, it is necessary to provide mathematical models that can accurately capture the underlying dynamics of medical needles. Here, we present a novel nonlinear mathematical model of flexible medical needles based on the Absolute Nodal Coordinate Formulation. The model allows the description of large needle deflections and arbitrarily large rigid body motions. Tailored to the requirements of transperineal prostate biopsy and brachytherapy, it can correlate both the translational and rotational coordinates of the needle's base with its deflection, provide force feedback and accept arbitrary loading conditions. The model is optimised in terms of computational efficiency in order to allow real-time simulation and control. Experiments show that the proposed model allows for submillimeter precision in both static and dynamic needle deflection settings. Due to its accuracy and computational efficiency, it is expected to constitute a valuable tool for both real-time visual/haptic simulation and control of percutaneous needle insertion.

This paper describes a novel tissue positioning system with an integrated suturing robot and demonstrates its ability to perform semi-automatic anastomoses of synthetic blood vessels. We began with a finite element analysis-based design consideration for achieving adequate grasping of blood vessels to demonstrate robust performance under expected clinical forces. We then conducted standardized positioning tests to measure the repeatability of the system and incorporated a high-resolution optical coherence tomography (OCT) fiber imaging sensor within the tip of the suturing tool to provide position feedback of the robot during a suturing task. Using the microvascular positioner and OCT sensor, the system performed semi-automatic suturing of synthetic 5 mm diameter blood vessels (\mathrm{N}=4\mathrm{N}=4), and the suture quality was evaluated for consistency in spacing, bite depth, percent lumen reduction, and maximum suture strength. The system completed the task in an average time of 31.75 minutes. The samples had zero missed stitches, average spacing of 1.64 mm, an average bite depth of 2.14 mm, an average lumen reduction of 57.98%, and an average suture strength of 3.13 N.

Changes in topological spatial relations of objects are often strong indicators for state transitions in the underlying processes they are involved in. While various aspects of semantic mapping have been extensively researched, the reasoning about the temporal development of spatial relations of instances is often neglected. This paper presents a concept to combine a semantic map with a stream processing framework for live analysis of the spatio-temporal relation of objects, based on the map and information inferred from sensors streams. To demonstrate the functionality of our concept, we implemented a proof-of-concept system to track everyday events in an office environment. The presented application scenario clearly demonstrates the benefits of the proposed architecture for detecting and handling complex spatio-temporal events.

Human-robot collaborative disassembly is an emerging trend in the sustainable recycling process of electronic and mechanical products. It requires the use of advanced technologies to assist workers in repetitive physical tasks and deal with creaky and potentially damaged components. Nevertheless, when disassembling worn-out or damaged components, unexpected robot behaviors may emerge, so harmless and symbiotic physical interaction with humans and the environment becomes paramount. This work addresses this challenge at the control level by ensuring safe and passive behaviors in unplanned interactions and contact losses. The proposed algorithm capitalizes on an energy-aware Cartesian impedance controller, which features energy scaling and damping injection, and an augmented energy tank, which limits the power flow from the controller to the robot. The controller is evaluated in a real-world flawed unscrewing task with a Franka Emika Panda and is compared to a standard impedance controller and a hybrid force-impedance controller. The results demonstrate the high potential of the algorithm in human-robot collaborative disassembly tasks.

Springs are commonly used in wearable robotic devices to provide assistive joint torque without the need for motors and batteries. However, different tasks (such as walking or running) and different users (such as athletes with strong legs or the elderly with weak legs) necessitate different assistive joint torques, and therefore, springs with different stiffness. Variable stiffness springs are a special class of springs which can exert more or less torque upon the same deflection, provided that the user is able to change the stiffness of the spring. In this paper, we present a novel variable stiffness spring design in which the user can select a preferred spring stiffness similar to switching gears on a bicycle. Using a leg-swing experiment, we demonstrate that the user can increment and decrement spring stiffness in a large range to effectively assist the hip joint during leg oscillations. Variable stiffness springs with human-selectable stiffness could be key components of wearable devices which augment locomotion tasks, such as walking, running, and swimming.

We introduce a spherical fingertip sensor for dynamic manipulation. It is based on barometric pressure and time-of-flight proximity sensors and is low-latency, compact, and physically robust. The sensor uses a trained neural network to estimate the contact location and three-axis contact forces based on data from the pressure sensors, which are embedded within the sensor's sphere of polyurethane rubber. The time-of-flight sensors face in three different outward directions, and an integrated microcontroller samples each of the individual sensors at up to 200 Hz. To quantify the effect of system latency on dynamic manipulation performance, we develop and analyze a metric called the collision impulse ratio and characterize the end-to-end latency of our new sensor. We also present experimental demonstrations with the sensor, including measuring contact transitions, performing coarse mapping, maintaining a contact force with a moving object, and reacting to avoid collisions.

Soft, wearable systems hold promise for a wide variety of new or enhanced applications in the realm of human-computer interaction, physiological monitoring, wear-able robotics, and a host of other human-centric devices. Soft sensor systems have been developed concurrently in order to allow these wearable systems to respond intelligently with their surroundings. A recently reported sensing mechanism based on the strain-mediated contact in anisotropically resistive structures (SCARS) is an attractive solution due to its high sensing resolution, low-profile nature, and high mechanical resilience. Furthermore, the resistance-based output provides a simple electronic readout, facilitating its use in a wide variety of applications. However, previous iterations of the sensing mech-anism have exhibited stress relaxation and hysteretic behaviors that limit the scope of its use. Here, we report an iteration of the SCARS mechanism that uses silicone-based materials with low mechanical loss in order to improve the sensor signal stability and bandwidth. A new fabrication approach is developed which permits the incorporation of a liquid elastomer adhesive layer while also preserving the SCARS sensing functionality. The silicone-based SCARS sensors exhibited fast stress relaxation response (< 1 s) and reduced cyclic drift properties by more than half that of previously reported designs. A physiological monitoring demonstration is presented, validating that the new sensor design is mechanically resilient to such applications and has potential for use in real-world wearable use cases.

This work presents the computational design and validation of the Multi-Arm Relocatable Manipulator (MARM), a three-limb robot for space applications, with particular reference to the MIRROR (i.e., the Multi-arm Installation Robot for Readying ORUs and Reflectors) use-case scenario as proposed by the European Space Agency. A holistic computational design and validation pipeline is proposed, with the aim of comparing different limb designs, as well as ensuring that valid limb candidates enable MARM to perform the complex loco-manipulation tasks required. Moti-vated by the task complexity in terms of kinematic reachability, (self)-collision avoidance, contact wrench limits, and motor torque limits affecting Earth experiments, this work leverages on multiple state-of-art planning and control approaches to aid the robot design and validation. These include sampling-based planning on manifolds, non-linear trajectory optimization, and quadratic programs for inverse dynamics computations with constraints. Finally, we present the attained MARM design and conduct preliminary tests for hardware validation through a set of lab experiments.

Applications of rolling diaphragm transmissions for medical and teleoperated robotics are of great interest, due to the low friction of rolling diaphragms combined with the power density and stiffness of hydraulic transmissions. However, the stiffness-enabling pressure preloads can form a tradeoff against bearing loading in some rolling diaphragm layouts, and transmission setup can be difficult. Utilization of cable drives compliment the rolling diaphragm transmission's advantages, but maintaining cable tension is crucial for optimal and consistent performance. In this paper, a coaxial opposed rolling diaphragm layout with cable drive and an electronic transmission control system are investigated, with a focus on system reliability and scalability. Mechanical features are proposed which enable force balancing, decoupling of transmission pressure from bearing loads, and maintenance of cable tension. Key considerations and procedures for automation of transmission setup, phasing, and operation are also presented. We also present an analysis of system stiffness to identify key compliance contributors, and conduct experiments to validate prototype design performance.

Designing an intuitive User Interface (UI) for controlling assistive robots remains challenging. Most existing UIs leverage traditional control interfaces such as joysticks, hand-held controllers, and 2D UIs. Thus, users have limited availability to use their hands for other tasks. Furthermore, although there is extensive research regarding legged manipulators, comparatively little is on their UIs. Towards extending the state-of-art in this domain, we provide a user study comparing an Augmented Reality (AR) Head-Mounted Display (HMD) UI we developed for controlling a legged manipulator against off-the-shelf control methods for such robots. We made this comparison baseline across multiple factors relevant to a successful interaction. The results from our user study (N=17N=17N=17) show that although the AR UI increases immersion, off-the-shelf control methods outperformed the AR UI in terms of time performance and cognitive workload. Nonetheless, a follow-up pilot study incorporating the lessons learned shows that AR UIs can outpace hand-held-based control methods and reduce the cognitive requirements when designers include hands-free interactions and cognitive offloading principles into the UI.

Transseptal puncture (TSP) is a prerequisite for left atrial catheter ablation for atrial fibrillation, requiring access from the right side of the heart. It is a demanding procedural step associated with complications, including inadvertent puncturing and application of large forces on the tissue wall. Robotic systems have shown great potential to overcome such challenges by introducing force-sensing capabilities and increased precision and localization accuracy. Therefore, this work introduces the design and development of a novel robotic system developed to perform TSP. We integrated optoelectronic sensors into the tools' fixtures, measuring tissue contact and puncture forces along one axis. The novelty of this design is in the system's ability to manipulate a Brockenbrough (BRK) needle and dilator-sheath simultaneously and measure tissue contact and puncture forces. In performing puncture experiments on anthropomorphic tissue models, an average puncture force of 3.97 ± 0.45 N (1SD) was established - similar to the force reported in literature on the manual procedure. This research highlights the potential for improving patient safety by enforcing force constraints, paving the way to more automated and safer TSP.

As soft robotic systems become increasingly complex, there is a need to develop sensory systems which can provide rich state information to the robot for feedback control. Multi-axis force sensing and control is one of the less explored problems in this domain. There are numerous challenges in the development of a multi-axis soft sensor: from the design and fabrication to the data processing and modelling. This work presents the design and development of a novel multi-axis soft sensor using a gelatin-based ionic hydrogel and 3D printing technology. A learning-based modelling approach coupled with sensor redundancy is developed to model the environmentally dependent soft sensors. Numerous real-time experiments are conducted to test the performance of the sensor and its applicability in closed-loop control tasks at 20 Hz. Our results indicate that the soft sensor can predict force values and orientation angle within 4% and 7% of their total range, respectively.

We propose a novel design for a lightweight and compact tunable stiffness actuator capable of stiffness changes up to 20x. The design is based on the concept of a coiled spring, where changes in the number of layers in the spring change the bulk stiffness in a near linear fashion. We present an elastica nested rings model for the deformation of the proposed actuator and empirically verify that the designed stiffness-changing spring abides by this model. Using the resulting model, we design a physical prototype of the tunable-stiffness coiled-spring actuator and discuss the effect of design choices on the resulting achievable stiffness range and resolution. In the future, this actuator design could be useful in a wide variety of soft robotics applications, where fast, controllable, and local stiffness change is required over a large range of stiffnesses.