IROS 2023 Papers

We present AVS-HRL, a modular Active visual SLAM system based on hierarchical reinforcement learning. The reward function explicitly considers the efficiency of exploration and the accuracy of mapping by utilizing the internal variables of SLAM, such as feature points distribution and loop-closure signal. Compared to end-to-end active SLAM methods, we designed a map reconstruction module that can correct the cumulative error in the incremental mapping process. Furthermore, the inputs of all neural network modules use more abstract and general information, such as grid maps, rather than raw sensor observations. We conducted experiments in two different simulators and real-world environments. In the noisy setting of Habitat environments, our method improves the accuracy of the mapped areas by 68.48% as an average of Gibson and MP3D datasets. Moreover, our method's generalization performance was demonstrated through direct transfer across different simulators and real-world environments.

Solving real-world manipulation tasks requires robots to be equipped with a repertoire of skills that can be applied to diverse scenarios. While learning-based methods can enable robots to acquire skills from interaction data, their success relies on collecting training data that covers the diverse range of tasks that the robot may encounter during the test time. However, creating diverse and feasible training tasks often requires extensive domain knowledge and non-trivial manual labor. We introduce Active Task Randomization (ATR), an approach that learns robust skills through the unsupervised generation of training tasks. ATR selects suitable training tasks-which consist of an environment configuration and manipulation goal-by actively balancing their diversity and feasibility. In doing so, ATR effectively creates a curriculum that gradually increases task diversity while maintaining a moderate level of feasibility, which leads to more complex tasks as the skills become more capable. ATR predicts task diversity and feasibility with a compact task representation that is learned concurrently with the skills. The selected tasks are then procedurally generated in simulation with a graph-based parameterization. We demonstrate that the learned skills can be composed by a task planner to solve unseen sequential manipulation problems based on visual inputs. Compared to baseline methods, ATR can achieve superior success rates in single-step and sequential manipulation tasks. Videos are available at sites.google.com/view/active-task-randomization/

In this work, we present a method to estimate the planar mass distribution of a rigid object through robotic interactions and force/torque feedback. This is a challenging problem because of the complexity of modeling physical dynamics and the action dependencies across the model parameters. We propose a sequential estimation strategy combined with a set of robot action selection rules based on the analytical formulation of a discrete-time dynamics model. To evaluate the performance of our approach, we also manufactured re-configurable block objects that allow us to modify the object mass distribution while having access to the ground truth values. We compare our approach against multiple baselines and show that it can estimate the mass distribution with around 10% error, while the baselines have errors ranging from 18% to 68%.

This paper describes the hardware implementation and characterization of a capacitive sensor designed to support detection and localization of nearby objects. The sensor can be mounted on the exterior of any given robotic system. The technology is particularly well-suited to detection of capacitive material, such as living tissue. As such, it offers perspectives of facilitating human-robot interactions (cobotics). We exploit experimental data to implement a digital model of the sensor and illustrate its accuracy by emulating experimental results in simulation. The sensor is used in a number of interaction scenarii (following and avoidance), providing examples of the manner in which it can be used to support human-robot interactions.

Recently, active research has been conducted on the therapeutic functions of capsule endoscopes. Here, we propose an active capsule system that captures images of the interior of the gastrointestinal tract (GI) and actively delivers therapeutic patches. The active capsule system mainly comprises therapeutic patches, an active capsule equipped with a camera, and a robot-assisted magnetic actuator. The active capsule moves inside the GI tract via a magnetic actuator using a robot, captures pictures of the GI tract in actual time, and performs hemostatic treatment by delivering therapeutic patches to the target lesions. First, the fundamental performance of the active capsule system was verified via a hemostatic performance test of the therapeutic patch, patch contamination prevention test of the active capsule, and basic actuation test of the capsule. Second, multiple therapeutic patches were delivered to the gastric surface in an ex vivo test using an active capsule system. Finally, as a preclinical test, it was confirmed that the GI tract examination and the therapeutic patches delivery were possible using the active capsule system through an animal test using a porcine. Consequently, the proposed active capsule system represents a new paradigm for capsule endoscopy with multiple therapeutic patch delivery capabilities.

Perception in robot manipulation has been actively explored with the goal of advancing and integrating vision and touch for global and local feature extraction. However, it is difficult to perceive certain object internal states, and the integration of visual and haptic perception is not compact and is easily biased. We propose to address these limitations by developing an active acoustic sensing method for robot manipulation. Active acoustic sensing relies on the resonant properties of the object, which are related to its material, shape, internal structure, and contact interactions with the gripper and environment. The sensor consists of a vibration actuator paired with a piezo-electric microphone. The actuator generates a waveform, and the microphone tracks the waveform's propagation and distortion as it travels through the object. This paper presents the sensing principles, hardware design, simulation development, and evaluation of physical and simulated sensory data under different conditions as a proof-of-concept. This work aims to provide fundamentals on a useful tool for downstream robot manipulation tasks using active acoustic sensing, such as object recognition, grasping point estimation, object pose estimation, and external contact formation detection.

Obtaining high-density tactile field information is a critical aspect of research in the field of robotic haptics, as it plays a decisive role in determining the precision of robot manipulations. Vision-based tactile sensors have unique high-resolution features, which make them promising for related research. However, previous studies have mainly focused on reconstructing the shape of rigid objects or predicting the three-dimensional force of rigid objects, neglecting the analysis of flexible objects. Moreover, due to the resolution limitations of existing commercial sensors, the performance evaluation of previous force prediction models relied solely on the total force. To overcome these limitations and in order to explore the tactile field information of objects with more attributes, this paper presents a detailed high-density tactile field data acquisition method based on a mechanical simulation environment. Additionally, we constructed a network to learn the mapping relationship between tactile images and six-dimensional tactile field information. Our results demonstrate that the proposed method can predict the three-dimensional force and displacement information of the object. Notably, the prediction error is within the tolerance range for fine manipulation by robots.

Contact sensing and localization capabilities that mimic human skin are highly desirable for robots. In this paper, we introduce AcouSkin, an acoustic wave based full surface contact localization system. Acoustic waves produced by piezoelectric transceivers using a monotone are coupled to surfaces turning them into an active sensor. Our system leverages information from four piezoelectric transceivers mounted on the surface of an acrylic sheet and vacuum cleaner robot bumper to localize contacts to 18 unique segments. We first characterize acoustic wave propagation based on signal and material properties and then propose hardware and software methods to realize full surface contact localization. Our results show that AcouSkin can reliably localize contact on a flat acrylic sheet with 18 uniformly spaced locations across a 54cm length with mean absolute error (MAE) of ≤ 1 locations using maximum likelihood estimator (MLE) and multilayer perceptron (MLP) models. On the vacuum cleaner robot bumper AcouSkin shows a zero MAE. Further, the system is also able to localize contacts made using forces as low as 2N (Newtons) and as high as 20N. Overall, AcouSkin provides full surface contact localization while requiring minimal instrumentation with easy deployment on real-world robots.

Visual-Inertial (VI) sensors are popular in robotics, self-driving vehicles, and augmented and virtual reality applications. In order to use them for any computer vision or state-estimation task, a good calibration is essential. However, collecting informative calibration data in order to render the calibration parameters observable is not trivial for a non-expert. In this work, we introduce a novel VI calibration pipeline that guides a non-expert with the use of a graphical user interface and information theory in collecting informative calibration data with Next-Best-View and Next-Best-Trajectory suggestions to calibrate the intrinsics, extrinsics, and temporal misalignment of a VI sensor. We show through experiments that our method is faster, more accurate, and more consistent than state-of-the-art alternatives. Specifically, we show how calibrations with our proposed method achieve higher accuracy estimation results when used by state-of-the-art VI Odometry as well as VI-SLAM approaches. The source code of our software can be found on: https://github.com/chutsu/yac.

Soft robotics requires effective tools to educate the next generation of engineers and researchers. Stemming from a lack of universally accepted principles for education and with high barriers to entry in terms of fabrication and hardware, education to date has been highly ad hoc. We present a low-cost toolkit based on re-configurable balloon which allows rapid development of soft yet functional robots. This provides practical demonstrations of key soft robotic principles including: morphology, stiffness control, controller dependencies and modulation of environmental interactions, while grounding robot behaviours in fundamental mechani-cal models. We provide a framework for assembling balloon structures, incorporating actuation and exploring interactions. A diverse set of robots have been developed to show the potential to use this balloon-bots for educational activities for undergraduate teaching or below. In particular, different modes of locomotion are shown using robots each of which has an assembly time under 5 minutes. These robots can teach skills ranging from component integration and implementation, to key soft robotic design principles and embodied intelligence.

Efficient and effective exploration in continuous space is a central problem in applying reinforcement learning (RL) to autonomous driving. Skills learned from expert demonstrations or designed for specific tasks can benefit the exploration, but they are usually costly-collected, unbalanced/suboptimal, or failing to transfer to diverse tasks. However, human drivers can adapt to varied driving tasks without demonstrations by taking efficient and structural explorations in the entire skill space rather than a limited space with task-specific skills. Inspired by the above fact, we propose an RL algorithm exploring all feasible motion skills instead of a limited set of task-specific and object-centric skills. Without demonstrations, our method can still perform well in diverse tasks. First, we build a task-agnostic and ego-centric (TaEc) motion skill library in a pure motion perspective, which is diverse enough to be reusable in different complex tasks. The motion skills are then encoded into a low-dimension latent skill space, in which RL can do exploration efficiently. Validations in various challenging driving scenarios demonstrate that our proposed method, TaEc-RL, outperforms its counterparts significantly in learning efficiency and task performance.

Fiducial markers are important tools for robotic navigation and imaging, enabling accurate localization and tracking of objects in challenging environments. In this paper, we present AcTag, a new fiducial marker design for use underwater with imaging sonar and cameras, as well as a method for the detection of AcTags within acoustic images. High amounts of noise and a nonlinear projection model make it difficult to use imaging sonar in autonomous localization and mapping. In order to expand the use of imaging sonar in autonomous underwater vehicles, our markerb design and detection algorithm for sonar images facilitate the identification of four unique landmarks per tag, and provide relative range and azimuth values to each landmark. We evaluate our marker and detection algorithm with simulated and real-world sonar data, reporting on the false positive and true positive rates, as well as the estimated error for the range and azimuth estimates per landmark. We also release an open-source library for generating tag families and detecting the tags.

In this study, we introduce AV-PedAware, a self-supervised audio-visual fusion system designed to improve dynamic pedestrian awareness for robotics applications. Pedestrian awareness is a critical requirement in many robotics applications. However, traditional approaches that rely on cameras and LIDARs to cover multiple views can be expensive and susceptible to issues such as changes in illumination, occlusion, and weather conditions. Our proposed solution replicates human perception for 3D pedestrian detection using low-cost audio and visual fusion. This study represents the first attempt to employ audio-visual fusion to monitor footstep sounds for the purpose of predicting the movements of pedestrians in the vicinity. The system is trained through self-supervised learning based on LIDAR-generated labels, making it a cost-effective alternative to LIDAR-based pedestrian awareness. AV-PedAware achieves comparable results to LIDAR-based systems at a fraction of the cost. By utilizing an attention mechanism, it can handle dynamic lighting and occlusions, overcoming the limitations of traditional LIDAR and camera-based systems. To evaluate our approach's effectiveness, we collected a new multimodal pedestrian detection dataset and conducted experiments that demonstrate the system's ability to provide reliable 3D detection results using only audio and visual data, even in extreme visual conditions. We will make our collected dataset and source code available online for the community to encourage further development in the field of robotics perception systems.

Generating natural and physically feasible motions for legged robots has been a challenging problem due to its complex dynamics. In this work, we introduce a novel learning-based framework of autoregressive motion planner (ARMP) for quadruped locomotion and navigation. Our method can generate motion plans with an arbitrary length in an autore-gressive fashion, unlike most offline trajectory optimization algorithms for a fixed trajectory length. To this end, we first construct the motion library by solving a dense set of trajectory optimization problems for diverse scenarios and parameter settings. Then we learn the motion manifold from the dataset in a supervised learning fashion. We show that the proposed ARMP can generate physically plausible motions for various tasks and situations. We also showcase that our method can be successfully integrated with the recent robot navigation frameworks as a low-level controller and unleash the full capability of legged robots for complex indoor navigation.

Utilizing wind hovering techniques of soaring birds can save energy expenditure and improve the flight endurance of micro air vehicles (MAVs). Here, we present a novel method for fully autonomous orographic soaring without a priori knowledge of the wind field. Specifically, we devise an Incremental Nonlinear Dynamic Inversion (INDI) controller with control allocation, adapting it for autonomous soaring. This allows for both soaring and the use of the throttle if necessary, without changing any gain or parameter during the flight. Furthermore, we propose a simulated-annealing-based optimization method to search for soaring positions. This enables for the first time an MAV to autonomously find a feasible soaring position while minimizing throttle usage and other control efforts. Autonomous orographic soaring was performed in the wind tunnel. The wind speed and incline of a ramp were changed during the soaring flight. The MAV was able to perform autonomous orographic soaring for flight times of up to 30 minutes. The mean throttle usage was only 0.25% for the entire soaring flight, whereas normal powered flight requires 38%. Also, it was shown that the MAV can find a new soaring spot when the wind field changes during the flight.

Humans have latency in their visual perception system between observation and action. Any action we take is based on an earlier observation since, by the time we act, the state has already changed, and we got a new observation. In autonomous driving, this latency is also present, determined by the amount of time the control algorithm needs to process information before acting. This algorithmic perception latency can be reduced by massive computing power via GPUs and FPGAs, which is improbable in automobile platforms. Thus, it is a reasonable assumption that the algorithmic perception latency is inevitable. Many researchers have developed different neural network driving models without consideration of the algorithmic perception latency. This paper studies the latency effect on vision-based neural network autonomous driving in the lane-keeping task and proposes a vision-based novel neural network controller, the Adaptive Neural Ensemble Controller (ANEC) that is inspired by the near/far gaze distribution of human drivers during lane-keeping. ANEC was tested in Gazebo 3D simulation environment with Robot Operating System (ROS) which showed the effectiveness of ANEC in dealing with algorithmic latency. The source code is available at https://github.com/jrkwon/oscar/tree/devel_anec.

We propose a lightweight, effective, and efficient anti-drone network, namely ADMNet, for visually detecting and monitoring unfriendly drones with a constrained view field, flying against a complex environment. We merge an SPP module to the first head of YOLOv4 to improve accuracy and perform network compression to reduce inference latency and model size. To compensate for the accuracy loss caused by condensation, we propose an SPPS module and a ResNeck module for the neck of the network and implement an effective attention module for the backbone. Eventually, we present an accurate and compact ADMNet with barely 3.9 MB, ensuring low computational cost and real-time detection. Our method achieves state-of-the-art performance on three challenging real-world datasets (Average Precision @0.5IoU): Det-Fly 96.2%, NPS-Drones 92.0%, and TIBNet 89.7%. The throughput is higher than the prior work, in addition to its superior performance. The comparative testing in real-world scenarios proves that our method exhibits strong reliability and generalization ability. Deploying the network on drone onboard edge-computing devices enables real-time detection and monitoring of flying drones, highlighting the portability and viability of the ADMNet.

There is increasing evidence of the role of compromised mediolateral balance in falls and the need for rehabilitation specifically focused on mediolateral direction for various populations with motor deficits. To address this need, we have developed a neurorehabilitation platform by integrating a wearable robotic hip abduction-adduction exoskeleton with a visual interface. The platform is expected to influence and rehabilitate the underlying visuomotor mechanisms in individuals by having users perform motion tasks based on visual feedback while the robot applies various controlled resistances governed by the admittance controller implemented in the robot. A preliminary study was performed on 3 non disabled individuals to analyze the performance of the system and observe any adaptation in hip joint kinematics and kinetics as a result of the visuomotor training under 4 different admittance conditions. All three subjects exhibited increased consistency of motion during training and interlimb coordination to achieve motion tasks, demonstrating the utility of the system. Further analysis of observed human-robot torque interactions and electromyography (EMG) signals, and its implication in neurorehabilitation aimed at populations suffering from chronic stroke are discussed.

This paper introduces and solves a visibility-based escort planning problem. This novel problem, which is closely related to the well-researched family of visibility-based pursuit-evasion problems in robotics, entails an escort agent tasked with escorting a vulnerable agent, called the VIP, in a 2-dimensional environment. The escort protects the VIP from adversaries that pose line-of-sight threats. We describe a correct and complete planning algorithm whose inputs are a simply-connected polygonal map of the environment, starting locations for the escort and the VIP, along with a goal location to which the VIP agent should be safely moved. The algorithm computes trajectories for the escort and VIP which allow the VIP to reach its goal without coming into the line-of-sight of the adversary at any time. During the execution of these trajectories, the adversary is allowed to move along any continuous path that does not enter into the line-of-sight of the escort. The algorithm proceeds by dividing the environment into a collection of conservative regions and planning the escort's movements as a sequence of these regions via breadth-first search over an information graph. The trajectory of the VIP can then be constructed by tracing the ‘safe zones' swept out by the escort's trajectory. We describe an implementation of this algorithm and present computed examples of escort agent strategies in diverse environments.