IROS 2023 Papers

The aim of this research is to investigate the relationship between spinal flexion and quadruped locomotion in a rat robot equipped with a compliant spine, controlled by a central pattern generator (CPG). The study reveals that spinal flexion can enhance limb stride length, but it may also cause significant and unexpected motion disturbances during stride length variations. To address this issue, this paper proposes a CPG model driven by spinal flexion and a novel oscillator that incorporates a circular limit cycle and accounts for the anticipated stride length transition process. This approach effectively matches the torque change with the dynamics of stride length changes, leading to lower energy consumption. Extensive simulations are conducted to evaluate the efficacy of the proposed oscillator and compare it with the original kinetic model and other CPG models. The results demonstrate that the designed CPG model with the proposed oscillator yields smoother gait transitions during stride length variations and reduces energy consumption.

Efficient and accurate 3D object shape reconstruction contributes significantly to the success of a robot's physical interaction with its environment. Acquiring accurate shape information about unknown objects is challenging, especially in unstructured environments, e.g. the vision sensors may only be able to provide a partial view. To address this issue, tactile sensors could be employed to extract local surface information for more robust unknown object shape estimation. In this paper, we propose a novel approach for efficient unknown 3D object shape exploration and reconstruction using a multi-fingered hand equipped with tactile sensors and a depth camera only providing a partial view. We present a multi-finger sliding touch strategy for efficient shape exploration using a Bayesian Optimization approach and a single-leader-multi-follower strategy for multi-finger smooth local surface perception. We evaluate our proposed method by estimating the 3D shape of objects from the YCB and OCRTOC datasets based on simulation and real robot experiments. The proposed approach yields successful reconstruction results relying on only a few continuous sliding touches. Experimental results demonstrate that our method is able to model unknown objects in an efficient and accurate way.

Automating the process of fleece contaminant removal has the potential to drastically improve the quality of wool leaving the farm gate. Towards this goal, we present a method to automatically extract skirting lines, i.e., the separations between clean and contaminated wool of a fleece using RGB images. We propose a learning-based sparse-to-dense approach for estimating the non-rigid deformation of fleeces in order to estimate the skirting lines. Our method is bootstrapped from a set of sparse inlier feature correspon-dences, which are heavily filtered through a set of strict criteria. The inlier correspondences are then greedily expanded by adding correspondences from a denser set through a filtering process. This process is based on a learning approach that takes as inputs the pixel similarity and the consistency with their inlier neighbours. Each greedy iteration is initialised with a non-rigid deformation using as-rigid-as-possible as a prior to the filtering process. The proposed method outperforms both a rigid deformation baseline and optic flow deep learning approach, as evidenced by the quantitative evaluation of pixel location error in controlled experiments. To further prove its practicality, we demonstrate qualitative results comparing the predicted skirting line from various methods on images of skirted fleeces collected from several wool sheds.

It is imperative that robots can understand natural language commands issued by humans. Such commands typically contain verbs that signify what action should be performed on a given object and that are applicable to many objects. We propose a method for generalizing manipulation skills to novel objects using verbs. Our method learns a probabilistic classifier that determines whether a given object trajectory can be described by a specific verb. We show that this classifier accurately generalizes to novel object categories with an average accuracy of 76.69% across 13 object categories and 14 verbs. We then perform policy search over the object kinematics to find an object trajectory that maximizes classifier prediction for a given verb. Our method allows a robot to generate a trajectory for a novel object based on a verb, which can then be used as input to a motion planner. We show that our model can generate trajectories that are usable for executing five verb commands applied to novel instances of two different object categories on a real robot.

The need for autonomous robot systems in both the service and the industrial domain is larger than ever. In the latter, the transition to small batches or even “batch size 1” in production created a need for robot control system architectures that can provide the required flexibility. Such architectures must not only have a sufficient knowledge integration framework. It must also support autonomous mission execution and allow for interchangeability and interoperability between different tasks and robot systems. We introduce SkiROS2, a skill-based robot control platform on top of ROS. SkiROS2 proposes a layered, hybrid control structure for automated task planning, and reactive execution, supported by a knowledge base for reasoning about the world state and entities. The scheduling formulation builds on the extended behavior tree model that merges task-level planning and execution. This allows for a high degree of modularity and a fast reaction to changes in the environment. The skill formulation based on pre-, hold-and post-conditions allows to organize robot programs and to compose diverse skills reaching from perception to low-level control and the incorporation of external tools. We relate SkiROS2 to the field and outline three example use cases that cover task planning, reasoning, multisensory input, integration in a manufacturing execution system and reinforcement learning.

Soft Robotics has established itself as an integral field in the broader discipline of general robotics through multiple advantages like inherent safety, adaptable morphology, and energy- and weight efficiency. Especially in environments hostile to humans and classical robots like the deep sea, soft robotic structures made out of silicone and actuated by seawater have numerous advantages. An application with a huge scientific and commercial potential for soft robotic solutions is suction sampling for marine geology in depths of up to 6000 m. In this paper, we propose a single channel soft robotic actuator that is able to bend into six directions while absorbing process forces. By embedding a low melting point alloy (LMPA) acting as switchable strain-limiting structures, the actuator is capable of hexa-planar bending of up to 40° and elongation of 30 % with only one valve used for actuation. In addition, the LMPA chambers enable a stiffening factor of 4.1 and locking the actuator in its bending state for energy efficient usage in robotic deep-sea suction sampling.

As the future of autonomous underwater vehicle (AUV) deployments tends to multi-vehicle systems, new approaches in coordination and control are needed. In this work, we consider the problem of simultaneous survey and inspection where one vehicle dynamically discovers objects while another vehicle must inspect as many of the objects as possible over the course of the mission. This requires a fully autonomous inspection vehicle, and to this end, we present a planning approach which couples sampling-based motion planning with timed roadmap constraints as well as a real-time execution framework. The methods presented address the underlying challenges that arise during simultaneous survey and inspection using AUVs, namely those of communication constraints, safety of navigation constraints, and dynamically discovered tasks. Additionally, we present field results for the simultaneous survey and inspection mission using teamed AUVs.

In this paper, we address task and motion plan-ning (TAMP) which is an important yet challenging robotics problem. It is known to suffer from the high combinatorial complexity of discrete search, often requiring a large number of geometric planning calls. We build upon recent works in TAMP by taking advantage of learning methods to provide action feasibility information as a heuristic to the symbolic planner, thus guiding it to a geometrically feasible solution and reducing geometric planning time. We propose AGFP-Net, a multi-task neural network predicting not only action feasibility, but also the feasibility of a set of grasp types. We also propose an improved feasibility-informed TAMP algorithm capable of solving more complex problems, and handling goals which are not fully specified. Comparative results obtained on different problems of varying complexity show that our method is able to greatly reduce task and motion planning time.

Nano-quadcopters are versatile platforms attracting the interest of both academia and industry. Their tiny form factor, i.e., ~ 10 cm diameter, makes them particularly useful in narrow scenarios and harmless in human proximity. However, these advantages come at the price of ultra-constrained onboard computational and sensorial resources for autonomous operations. This work addresses the task of estimating human pose aboard nano-drones by fusing depth and images in a novel CNN exclusively trained in simulation yet capable of robust predictions in the real world. We extend a commercial off-the-shelf (COTS) Crazyflie nano-drone - equipped with a 320x240 px camera and an ultra-low-power System-on-Chip - with a novel multi-zone (8 x 8) depth sensor. We design and compare different deep-learning models that fuse depth and image inputs. Our models are trained exclusively on simulated data for both inputs, and transfer well to the real world: field testing shows an improvement of 58% and 51 % of our depth+camera system w.r.t. a camera-only State-of-the-Art baseline on the horizontal and angular mean pose errors, respectively. Our prototype is based on COTS components, which facilitates reproducibility and adoption of this novel class of systems.

Simultaneously controlling multiple robot swarms is challenging for a single human operator. When involving multiple operators, however, they can each focus on controlling a specific robot swarm, which helps distribute the cognitive workload. They could also exchange some robots with each other in response to the requirements of the tasks they discover. This paper investigates the ability of multiple operators to dynamically share the control of robot swarms and the effects of different communication types on performance and human factors. A total of 52 participants completed an experiment in which they were randomly paired to form a team. In a 2\times 22\times 2 mixed factorial study, participants were split into two groups by communication type (direct vs. indirect). Both groups experienced different robot-sharing conditions (robot-sharing vs. no-robot-sharing). Results show that although the ability to share robots did not necessarily increase task scores, it allowed the operators to switch between working independently and collaboratively, reduced the total energy consumed by the swarm, and was considered useful by the participants.

Shared-autonomy control strategies in teleoperation combine human decision-making and robot precision to solve complex tasks. In other words, advanced autonomous control algorithms can compensate for imprecise human commands, reduce the mental workload of the user, and enable the execution of tasks that otherwise wouldn't be feasible. This paper addresses one of these previously challenging scenarios. Herein, we present a novel control framework and motion generator that allows for real-time non-prehensile slosh-free teleoperation of liquids. The proposed approach is able to generate robust trajectories on the follower side which ensures task-space, joint-space, and manipulability constraint satisfaction. Our findings were evaluated through user studies and real-world scenarios. Participants were even explicitly challenged to try to spill liquid through teleoperation, reaching speeds up to 0.6 m/s.

In this paper, we propose a physics-based robot controller to deform a soft object toward a desired 3D shape using a limited number of handling points. For this purpose, the shape of the deformable object is represented using Fourier descriptors. We derive the analytical relation that provides the variation of the Fourier coefficients as a function of the movements of the handling points by considering a mass-spring model (MSM). A control law is then designed from this relation. Since the MSM provides an approximation of the object behavior, which in practice can lead to a drift between the object and its model, an online realignment of the model with the real object is performed by tracking its surface from data provided by a remote RGB-D camera. Simulation results validate the approach for the case where many points interact on a 2D soft object while experimental results obtained with two robotic arms demonstrate the autonomous shaping of a 3D soft object.

In this paper, we present a novel clothoid-based visual servoing method for controlling the shape of a variable length continuum manipulator. Clothoids are curves with linearly changing curvature. They allow us to obtain a smooth representation of a continuum manipulator's shape in a compact form with few parameters. Using this curve model, we generate image features that are used in an adaptive visual servoing method to drive the robot to a desired shape. The adaptive algorithm estimates and updates a local interaction matrix that maps the rate of change in clothoid features to actuator velocities of the continuum manipulator. As such, the method does not require any robot model or even actuator encoder measurements and only uses the visual clothoid features to control the robot shape. A unique advantage of using our clothoid representation is being able to generate reference shape curves without the need for taking images of the robot at the desired shapes. Experiments demonstrate successful shape and end effector pose convergence for a diverse set of references. Our repeatability tests demonstrate that the system performance is consistent. Notably, we also present the first results in the literature for the vision-based shape control of a variable length continuum robot, extending and contracting to achieve the desired shape.

Shape completion, i.e., predicting the complete geometry of an object from a partial observation, is highly relevant for several downstream tasks, most notably robotic manipulation. When basing planning or prediction of real grasps on object shape reconstruction, an indication of severe geometric uncertainty is indispensable. In particular, there can be an irreducible uncertainty in extended regions about the presence of entire object parts when given ambiguous object views. To treat this important case, we propose two novel methods for predicting such uncertain regions as straightforward extensions of any method for predicting local spatial occupancy, one through postprocessing occupancy scores, the other through direct prediction of an uncertainty indicator. We compare these methods together with two known approaches to probabilistic shape completion. Moreover, we generate a dataset, derived from ShapeNet [1], of realistically rendered depth images of object views with ground-truth annotations for the uncertain regions. We train on this dataset and test each method in shape completion and prediction of uncertain regions for known and novel object instances and on synthetic and real data. While direct uncertainty prediction is by far the most accurate in the segmentation of uncertain regions, both novel methods outperform the two baselines in shape completion and uncertain region prediction, and avoiding the predicted uncertain regions increases the quality of grasps for all tested methods. Web: https://github.com/DLR-RM/shape-completion

There are two major challenges for scaling up robot navigation around dynamic obstacles: the complex interaction dynamics of the obstacles can be hard to model analytically, and the complexity of planning and control grows exponentially in the number of obstacles. Data-driven and learning-based methods are thus particularly valuable in this context. However, data-driven methods are sensitive to distribution drift, making it hard to train and generalize learned models across different obstacle densities. We propose a novel method for compositional learning of Sequential Neural Control Barrier models (SN-CBFs) to achieve scalability. Our approach exploits an important observation: the spatial interaction patterns of multiple dynamic obstacles can be decomposed and predicted through temporal sequences of states for each obstacle. Through decomposition, we can generalize control policies trained only with a small number of obstacles, to environments where the obstacle density can be 100x higher. We demonstrate the benefits of the proposed methods in improving dynamic collision avoidance in comparison with existing methods including potential fields, end-to-end reinforcement learning, and model-predictive control. We also perform hardware experiments and show the practical effectiveness of the approach in the supplementary video.

We investigate the sequential manipulation planning problem for unmanned aerial manipulators (UAMs). Unlike prior work that primarily focuses on one-step manipulation tasks, sequential manipulations require coordinated motions of a UAM's floating base, the manipulator, and the object being manipulated, entailing a unified kinematics and dynamics model for motion planning under designated constraints. By leveraging a virtual kinematic chain (VKC)-based motion planning framework that consolidates components' kinematics into one chain, the sequential manipulation task of a UAM can be planned as a whole, yielding more coordinated motions. Integrating the kinematics and dynamics models with a hierarchical control framework, we demonstrate, for the first time, an over-actuated UAM achieves a series of new sequential manipulation capabilities in both simulation and experiment.

A useful capability is that of classifying some agent's behavior using data from a sequence, or trace, of sensor measurements. The sensor selection problem involves choosing a subset of available sensors to ensure that, when generated, observation traces will contain enough information to determine whether the agent's activities match some pattern. In generalizing prior work, this paper studies a formulation in which multiple behavioral itineraries may be supplied, with sensors selected to distinguish between behaviors. This allows one to pose fine-grained questions, e.g., to position the agent's activity on a spectrum. In addition, with multiple itineraries, one can also ask about choices of sensors where some behavior is always plausibly concealed by (or mistaken for) another. Using sensor ambiguity to limit the acquisition of knowledge is a strong privacy guarantee, a form of guarantee which some earlier work examined under formulations distinct from our inter-itinerary conflation approach. By concretely formulating privacy requirements for sensor selection, this paper connects both lines of work in a novel fashion: privacy-where there is a bound from above, and behavior verification-where sensors choices are bounded from below. We examine the worst-case computational complexity that results from both types of bounds, proving that upper bounds are more challenging under standard computational complexity assumptions. The problem is intractable in general, but we introduce an approach to solving this problem that can exploit interrelationships between constraints, and identify opportunities for optimizations. Case studies are presented to demonstrate the usefulness and scalability of our proposed solution, and to assess the impact of the optimizations.

Telesurgery has a clear potential for providing high-quality surgery to medically underserved areas like rural areas, battlefields, and spacecraft; nevertheless, effective methods to overcome unreliable communication systems are still lacking. Furthermore, it is not well understood how users react at the moment of communication loss and also during the loss. In this paper, we aim to analyze human response by proposing a telesurgery simulation framework that models an environment incorporating local and remote sites. Furthermore, this framework generates structural data for human behavior analysis and can provide different forms of assistance during the communication failure and at the communication recovery. We investigated three different types of assistance: User-centered, Robot-centered and Hybrid. A 12-person user-study was carried out using the proposed telesurgery simulation where participants completed a peg transfer task with random communication loss. The collected data was used to analyze the human response to a communication failure. The proposed Hybrid method reduced temporal demand with no increase in completion time compared to the baseline control method where users were unable to move the input device during the communication loss. The Hybrid method also significantly reduced both the task completion time and workload compared to the other two proposed methods (User-centered and Robot-centered).

Robust mission planning is an essential component for mission autonomy to perform complicated tasks in extreme environments. In this paper, we are interested in the role of semantic abstractions for guiding autonomous mission planning. In particular, we focus on how semantics can be leveraged to transition, at the mission level, in between individually robust task plans. We present a mission autonomy framework wherein a task plan adaptation policy leverages up-to-date semantics information in order to adapt to changes that occur during run-time, which endows the robot with better resiliency to unexpected events and improves the overall efficiency of mission operations. Under this new perspective, we provide a concrete and challenging application of autonomous exploration and radio source seeking in a complex multi-level building environment. Experimental results over simulations and real hardware tests demonstrate that the presented semantics-aware mission adaptation more effectively completes the mission with better qualitative results compared to a non-adaptive baseline.