IROS 2023 论文列表

In the latest research for unmanned aerial vehicles (UAVs), time-optimal trajectory planning of a point-mass with acceleration as control input and constrained maximum velocity (TOT-PMAV) has proved to be very promising for UAV behavior planning. They can be calculated within microseconds and tracked with high precision by modern trajectory tracking controllers like model predictive control (MPC). However, recent research shows that the state-of-the-art (SOTA) approach to generate these time-optimal trajectories is based on an invalid method to synchronize the coordinate axes which sometimes yields trajectories that miss the desired final state by far. Hence, an alternative approach was proposed that claims to resolve this issue. However, it still needs mathematical proof of its correctness. In this work, we provide the missing proof and mathematically demonstrate the problems arising from the SOTA approach. Further, since neither the SOTA nor the alternative solution approach utilizes the full kinematic capacity of a UAV, we propose an improved solution approach to the TOT-PMAV that better exploits kinematic properties and yields, on average, up to 14% faster trajectories. We substantiate our findings with an extensive computational study, show in which situations the SOTA is likely to fail and provide metrics to measure the consequence during failure. To enable reproducibility, our code is open-source.

Microsurgery involves the dexterous manipulation of delicate tissue or fragile structures, such as small blood vessels and nerves, under a microscope. To address the limitations of imprecise manipulation of human hands, robotic systems have been developed to assist surgeons in performing complex microsurgical tasks with greater precision and safety. However, the steep learning curve for robot-assisted microsurgery (RAMS) and the shortage of well-trained surgeons pose significant challenges to the widespread adoption of RAMS. Therefore, the development of a versatile training system for RAMS is necessary, which can bring tangible benefits to both surgeons and patients. In this paper, we present a Tactile Internet-Based Micromanipulation System (TIMS) based on a ROS-Django web-based architecture for microsurgical training. This system can provide tactile feedback to operators via a wearable tactile display (WTD), while real-time data is transmitted through the internet via a ROS-Django framework. In addition, TIMS integrates haptic guidance to ‘guide’ the trainees to follow a desired trajectory provided by expert surgeons. Learning from demonstration based on Gaussian Process Regression (GPR) was used to generate the desired trajectory. We conducted user studies to verify the effectiveness of our proposed TIMS, comparing users' performance with and without tactile feedback and/or haptic guidance. For more details of this project, please view our website: https://sites.google.com/view/viewtims/home.

Predicting pedestrian motion is essential for developing socially-aware robots that interact in a crowded environment. While the natural visual perspective for a social interaction setting is an egocentric view, the majority of existing work in trajectory prediction therein has been investigated purely in the top-down trajectory space. To support first-person view trajectory prediction research, we present T2FPV, a method for constructing high-fidelity first-person view (FPV) datasets given a real-world, top-down trajectory dataset; we showcase our approach on the ETH/UCY pedestrian dataset to generate the egocentric visual data of all interacting pedestrians, creating the T2FPV-ETH dataset. In this setting, FPV-specific errors arise due to imperfect detection and tracking, occlusions, and field-of-view (FOV) limitations of the camera. To address these errors, we propose CoFE, a module that further refines the imputation of missing data in an end-to-end manner with trajectory forecasting algorithms. Our method reduces the impact of such FPV errors on downstream prediction performance, decreasing displacement error by more than 10% on average. To facilitate research engagement, we release our T2FPV-ETH dataset and software tools§§https://github.com/cmubig/T2FPV.

Deep perception models have to reliably cope with an open-world setting of domain shifts induced by different geographic regions, sensor properties, mounting positions, and several other reasons. Since covering all domains with annotated data is technically intractable due to the endless possible variations, researchers focus on unsupervised domain adaptation (UDA) methods that adapt models trained on one (source) domain with annotations available to another (target) domain for which only unannotated data are available. Current predominant methods either leverage semi-supervised approaches, e.g., teacher-student setup, or exploit privileged data, such as other sensor modalities or temporal data consistency. We introduce a novel domain adaptation method that leverages the best of both approaches. Our approach combines input data's temporal and cross-sensor geometric consistency with the mean teacher method. Dubbed T-UDA for “temporal UDA”, such a combination yields massive performance gains for the task of 3D semantic segmentation of driving scenes. Experiments are conducted on Waymo Open Dataset, nuScenes, and SemanticKITTI, for two popular 3D point cloud architectures, Cylinder3D and MinkowskiNet. Our codes are publicly available on https://github.com/ctu-vras/T-UDA.

In recent years, studies on Socially Assistive Robots (SARs) examine how to improve the quality of life of people living with dementia and older adults (OAs) in general. However, most SARs have somewhat limited perception capabilities or interact using simple pre-programmed responses, providing limited or repetitive interaction modalities. Integrating more advanced perceptual capabilities with deep learning processing would help move beyond such limitations. This paper presents T-Top, a tabletop robot designed with advanced audio and vision processing using deep learning neural networks. T-Top is made available as an open source platform with the goal of providing an experimental SAR platform that can implement richer interaction modalities with OAs.

In this paper, we address the problem of system identification and control of a front-steered vehicle which abides by the Ackermann geometry constraints. This problem arises naturally for on-road and off-road vehicles that require reliable system identification and basic feedback controllers for various applications such as lane keeping and way-point navigation. Traditional system identification requires expensive equipment and is time consuming. In this work we explore the use of differentiable physics for system identification and controller design and make the following contributions: i) We develop a differentiable physics simulator (DPS) to provide a method for the system identification of front-steered class of vehicles whose system parameters are learned using a gradient-based method; ii) We provide results for our gradient-based method that exhibit better sample efficiency in comparison to other gradient-free methods; iii) We validate the learned system parameters by implementing a feedback controller to demonstrate stable lane keeping performance on a real front-steered vehicle, the F1TENTH; iv) Further, we provide results exhibiting comparable lane keeping behavior for system parameters learned using our gradient-based method with lane keeping behavior of the actual system parameters of the F1TENTH.

We propose novel symmetry-based modeling and hybrid orientation-force control frameworks for cutaneous haptic device (CHD) to generate precise three degree-of-freedom (DoF) contact force on the fingertip robustly against user variability. The CHD hardware is designed in a form of an underactuated cable-driven parallel mechanism, with springs placed along the tendon to stabilize the pose. We analyze the kinematics of the CHD and propose a pose estimator by exploiting the symmetrical nature of the mechanism. We then devise a hybrid orientation-force controller to track the direction and magnitude of the desired contact force simultaneously in a feedback manner for control accuracy and robustness. We also adopt a tension regulator to mitigate friction effect during the actuation. Experimental validation and demonstration show the efficacy of the CHD with our proposed estimation and control framework.

In existing task and motion planning (TAMP) research, it is a common assumption that experts manually specify the state space for task-level planning. A well-developed state space enables the desirable distribution of limited computational resources between task planning and motion planning. However, developing such task-level state spaces can be non-trivial in practice. In this paper, we consider a long horizon mobile manipulation domain including repeated navigation and manipulation. We propose Symbolic State Space Optimization (S3O) for computing a set of abstracted locations and their 2D geometric groundings for generating task-motion plans in such domains. Our approach has been extensively evaluated in simulation and demonstrated on a real mobile manipulator working on clearing up dining tables. Results show the superiority of the proposed method over TAMP baselines in task completion rate and execution time.

This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. Given an instruction such as “Move the bottle on the left side of the plate to the empty chair,” the DSR is expected to identify the bottle and the chair from multiple candidates in the environment and carry the target object to the destination. Most of the existing multimodal language understanding methods are impractical in terms of computational complexity because they require inferences for all combinations of target object candidates and destination candidates. We propose Switching Head-Tail Funnel UNITER, which solves the task by predicting the target object and the destination individually using a single model. Our method is validated on a dataset based on a standard dataset for Vision-and-Language Navigation with object manipulation tasks. The results show that our method outperforms the baseline method in terms of language comprehension accuracy. Furthermore, we conduct physical experiments in which a DSR delivers standardized everyday objects in a standardized domestic environment as requested by instructions with referring expressions. The experimental results show that the object grasping and placing actions are achieved with success rates of more than 90 %.

In this paper, we propose a novel swashplateless-elevon actuation (SEA) for dual-rotor tail-sitter vertical takeoff and landing (VTOL) unmanned aerial vehicles (UAVs). In contrast to the conventional elevon actuation (CEA) which controls both pitch and yaw using elevons, the SEA adopts swash-plateless mechanisms to generate an extra moment through motor speed modulation to control pitch and uses elevons solely for controlling yaw, without requiring additional actuators. This decoupled control strategy mitigates the saturation of elevons' deflection needed for large pitch and yaw control actions, thus improving the UAV's control performance on trajectory tracking and disturbance rejection performance in the presence of large external disturbances. Furthermore, the SEA overcomes the actuation degradation issues experienced by the CEA when the UAV is in close proximity to the ground, leading to a smoother and more stable take-off process. We validate and compare the performances of the SEA and the CEA in various real-world flight conditions, including take-off, trajectory tracking, and hover flight and position steps under external disturbance. Experimental results demonstrate that the SEA has better performances than the CEA. Moreover, we verify the SEA's feasibility in the attitude transition process and fixed-wing-mode flight of the VTOL UAV. The results indicate that the SEA can accurately control pitch in the presence of high-speed incoming airflow and maintain a stable attitude during fixed-wing mode flight. Video of all experiments can be found in youtube.com/watch?v=Sx9Rk4Zf7sQ

Unlike most human-engineered systems, biological systems are emergent from low-level interactions, allowing much broader diversity and superior adaptation to complex environments. Inspired by the process of morphogenesis in nature, a bottom-up design approach for robot morphology is proposed to treat a robot's body as an emergent response to underlying processes rather than a predefined shape. This paper presents Loopy, a “Swarm-of-One” polymorphic robot testbed that can be viewed simultaneously as a robotic swarm and a single robot. Loopy's shape is determined jointly by self-organization and morphological computing using physically linked homogeneous cells. Experimental results show that Loopy can form symmetric shapes consisting of lobes. Using the same set of parameters, even small amounts of initial noise can change the number of lobes formed. However, once in a stable configuration, Loopy has an “inertia” to transfiguring in response to dynamic parameters. By making the connections among self-organization, morphological computing, and robot design, this work lays the foundation for more adaptable robot designs in the future.

Alginate hydrogels are widely researched in phar-maceutical applications for their abilities to encapsulate and dis-perse therapeutics in response to stimuli. While effective, their utility can be greatly improved once converted into artificial cell soft-microrobots, allowing them to actively navigate through complex in vivo environments and facilitate targeted drug deliv-ery. In this study, artificial cells were fabricated by crosslinking alginate with magnetic nanoparticles and then deployed within mucus solutions to characterize their propulsion capabilities. The goal of this study was to understand how variations in simplified gastrointestinal fluid, artificial cell properties, and magnetic field characteristics could affect surface locomotion. A comparison between automatic feedback control and manual “open-loop” operation was also quantitatively explored. Under feedback control, individual artificial cells were navigated with automatically generated waypoints and a PID controller. Simu-lations were used to verify controller performance and accuracy. User operation was carried out using an Xbox controller, where the joystick could directly change navigation direction. We conclude in this study that the surface navigation of artificial cells is highly predictable within mucus concentrations and that both feedback and open-loop control are equally successful in navigation.

Semantic segmentation, which aims to classify every pixel in an image, is a key task in machine perception, with many applications across robotics and autonomous driving. Due to the high dimensionality of this task, most existing approaches use local operations, such as convolutions, to generate per-pixel features. However, these methods are typically unable to effectively leverage global context information due to the high computational costs of operating on a dense image. In this work, we propose a solution to this issue by leveraging the idea of superpixels, an over-segmentation of the image, and applying them with a modern transformer framework. In particular, our model learns to decompose the pixel space into a spatially low dimensional superpixel space via a series of local cross-attentions. We then apply multi-head self-attention to the superpixels to enrich the superpixel features with global context and then directly produce a class prediction for each superpixel. Finally, we directly project the superpixel class predictions back into the pixel space using the associations between the superpixels and the image pixel features. Reasoning in the superpixel space allows our method to be substantially more computationally efficient compared to convolution-based decoder methods. Yet, our method achieves state-of-the-art performance in semantic segmentation due to the rich superpixel features generated by the global self-attention mechanism. Our experiments on Cityscapes and ADE20K demonstrate that our method matches the state of the art in terms of accuracy, while outperforming in terms of model parameters and latency.

In breast cancer patients, some nodules are only visible on MRI, thus, requiring MRI-guidance to perform the biopsy. MRI interventions are cumbersome due to the magnetic field and the constrained working space. An MR safe robotic system actuated by pneumatic stepper motors may enable these procedures, improving both accuracy and image-guided navigation. A compact multipurpose pneumatic stepper motor has been designed with outer dimensions (45 \times 40\times 15)\mathbf{mm}^{\mathbf{3}}(45 \times 40\times 15)\mathbf{mm}^{\mathbf{3}}. This is configurable as a linear, rotational or curved stepper motor with a customizable step size and radius of curvature. Five copies of these motors actuate the Sunram 7 biopsy robot, of which the moving part (without protruding racks and tubes) measures (130 \times 65\times 55)\mathbf{mm}^{\mathbf{3}}(130 \times 65\times 55)\mathbf{mm}^{\mathbf{3}}. After manually choosing the target location and angle of approach, the needle is robotically inserted into the breast and the integrated pneumatic biopsy gun is fired to sample tissue from the lesion. The maximum torque of the presented motor is 0.61 N m at 6 bar which can be achieved using 13-teeth polycarbonate gears. Using 17-teeth gears for higher accuracy and a more convenient working pressure of 2 bar the maximum torque is 0.28 N m. The accuracy in free air of the Sunram 7 robot is 1.69mm and 1.72mm in X and Z-direction respectively, with a resulting 2-D error of 2.54 mm. The workspace volume is 4.1 L. When targeting 10 mm-sized lesions in phantoms under MRI guidance, Sunram 7 achieved a success rate of 68%. The minimum interval between two successive biopsies was 5:47 minutes. The presented multipurpose stepper motor has distinct advantages over previous designs in terms of robustness, customizability, printability and ease of integration in MR safe robotics. The Sunram 7 is able to perform accurate MRI-guided biopsies in a large workspace volume while reducing the intervention time when compared to the gold standard (i.e., MRI-guided free-hand biopsy). Show More

The goal of intelligent embodied agents is to learn how to explore within the environment, interact with objects, and understand the environment in order to achieve task objectives. There are two main approaches to training such agents: one is to train an action policy that performs the task goal through end-to-end learning, and the other is to construct a policy by implementing the necessary abilities according to the task goal in a modular manner. For complex and long-horizon tasks, such as visual room rearrangement, a modular approach that infers task sequence by identifying the causality of actions through prior knowledge shows higher performance. Based on this insight, we propose an Online Subtask Prediction Network (OSPNet) that determines the subtask to be performed at each moment based on the environment information and past subtask inference history to train an embodied agent for long-horizon tasks through an end-to-end manner, and also propose a Subtask Aware Policy Network (SAPNet) as the action policy that decides actions based on the reasoning of the OSPNet. We implement an embodied agent that performs visual room rearrangement using the proposed SAPNet and train it through imitation learning, demonstrating similar or better performance with much fewer training steps than previous works.

We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially for categories not seen during training. By selecting informative interactions, Sf A discovers parts and reveals occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, Sf A accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a Sf A model trained in simulation can generalize to many unseen object categories with diverse structures and to real-world objects. Empirically, Sf A outperforms a pipeline of state-of-the-art components by 25.4 3D IoU percentage points on unseen categories, while matching already performant joint estimation baselines.11For code, data, and videos, see sfa.cs.columbia.edu/

We present a new robotic drawing system based on stroke-based rendering (SBR). Our motivation is the artistic quality of the whole performance. Not only should the generated strokes in the final drawing resemble the input image, but the stroke sequence should also exhibit a human artist's planning process. Thus, when a robot executes the drawing task, both the drawing results and the way the robot executes would look artistic. Our SBR system is based on image segmentation and depth estimation. It generates the drawing strokes in an order that allows for the intended shape to be perceived quickly and for its detailed features to be filled in and emerge gradually when observed by the human. This ordering represents a stroke plan that the drawing robot should follow to create an artistic rendering of images. We experimentally demonstrate that our SBR-based drawing makes visually pleasing artistic images, and our robotic system can replicate the result with proper sequences of stroke drawing.

Trajectory forecasting is a widely-studied problem for autonomous navigation. However, existing benchmarks evaluate forecasting based on independent snapshots of trajectories, which are not representative of real-world applications that operate on a continuous stream of data. To bridge this gap, we introduce a benchmark that continuously queries future trajectories on streaming data and we refer to it as “streaming forecasting.” Our benchmark inherently captures the disappearance and re-appearance of agents, presenting the emergent challenge of forecasting for occluded agents, which is a safetycritical problem yet overlooked by snapshot-based benchmarks. Moreover, forecasting in the context of continuous timestamps naturally asks for temporal coherence between predictions from adjacent timestamps. Based on this benchmark, we further provide solutions and analysis for streaming forecasting. We propose a plug-and-play meta-algorithm called “Predictive Streamer” that can adapt any snapshot-based forecaster into a streaming forecaster. Our algorithm estimates the states of occluded agents by propagating their positions with multi-modal trajectories, and leverages differentiable filters to ensure temporal consistency. Both occlusion reasoning and temporal coherence strategies significantly improve forecasting quality, resulting in 25% smaller endpoint errors for occluded agents and 10-20% smaller fluctuations of trajectories. Our work is intended to generate interest within the community by highlighting the importance of addressing motion forecasting in its intrinsic streaming setting. Code is available at https://github.com/ziqipang/StreamingForecasting.

Robust feature matching forms the backbone for most Visual Simultaneous Localization and Mapping (vSLAM), visual odometry, 3D reconstruction, and Structure from Motion (SfM) algorithms. However, recovering feature matches from texture-poor scenes is a major challenge and still remains an open area of research. In this paper, we present a Stereo Visual Odometry (StereoVO) technique based on point and line features which uses a novel feature-matching mechanism based on an Attention Graph Neural Network that is designed to perform well even under adverse weather conditions such as fog, haze, rain, and snow, and dynamic lighting conditions such as nighttime illumination and glare scenarios. We perform experiments on multiple real and synthetic datasets to validate our method's ability to perform StereoVO under low-visibility weather and lighting conditions through robust point and line matches. The results demonstrate that our method achieves more line feature matches than state-of-the-art line-matching algorithms, which when complemented with point feature matches perform consistently well in adverse weather and dynamic lighting conditions.