Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots

Halil Ibrahim Ugurlu; Xuan Huy Pham; Erdal Kayacan

doi:10.3390/robotics11050109

Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots

Halil Ibrahim Ugurlu, Xuan Huy Pham, Erdal Kayacan^*

^*Corresponding author for this work

Department of Electrical and Computer Engineering - Signal Processing and Machine learning

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

10 Citations (Scopus)

Abstract

In this study, a novel end-to-end path planning algorithm based on deep reinforcement learning is proposed for aerial robots deployed in dense environments. The learning agent finds an obstacle-free way around the provided rough, global path by only depending on the observations from a forward-facing depth camera. A novel deep reinforcement learning framework is proposed to train the end-to-end policy with the capability of safely avoiding obstacles. The Webots open-source robot simulator is utilized for training the policy, introducing highly randomized environmental configurations for better generalization. The training is performed without dynamics calculations through randomized position updates to minimize the amount of data processed. The trained policy is first comprehensively evaluated in simulations involving physical dynamics and software-in-the-loop flight control. The proposed method is proven to have a 38% and 50% higher success rate compared to both deep reinforcement learning-based and artificial potential field-based baselines, respectively. The generalization capability of the method is verified in simulation-to-real transfer without further training. Real-time experiments are conducted with several trials in two different scenarios, showing a 50% higher success rate of the proposed method compared to the deep reinforcement learning-based baseline.

Original language	English
Article number	109
Journal	Robotics
Volume	11
Issue	5
ISSN	2218-6581
DOIs	https://doi.org/10.3390/robotics11050109
Publication status	Published - Oct 2022

Keywords

deep reinforcement learning
obstacle avoidance
quadrotors
sim-to-real transfer

Access to Document

10.3390/robotics11050109Licence: CC BY

Cite this

@article{e51fb6036e644cfea6f39aca491c3ddb,

title = "Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots",

abstract = "In this study, a novel end-to-end path planning algorithm based on deep reinforcement learning is proposed for aerial robots deployed in dense environments. The learning agent finds an obstacle-free way around the provided rough, global path by only depending on the observations from a forward-facing depth camera. A novel deep reinforcement learning framework is proposed to train the end-to-end policy with the capability of safely avoiding obstacles. The Webots open-source robot simulator is utilized for training the policy, introducing highly randomized environmental configurations for better generalization. The training is performed without dynamics calculations through randomized position updates to minimize the amount of data processed. The trained policy is first comprehensively evaluated in simulations involving physical dynamics and software-in-the-loop flight control. The proposed method is proven to have a 38% and 50% higher success rate compared to both deep reinforcement learning-based and artificial potential field-based baselines, respectively. The generalization capability of the method is verified in simulation-to-real transfer without further training. Real-time experiments are conducted with several trials in two different scenarios, showing a 50% higher success rate of the proposed method compared to the deep reinforcement learning-based baseline.",

keywords = "deep reinforcement learning, obstacle avoidance, quadrotors, sim-to-real transfer",

author = "Ugurlu, {Halil Ibrahim} and Pham, {Xuan Huy} and Erdal Kayacan",

year = "2022",

month = oct,

doi = "10.3390/robotics11050109",

language = "English",

volume = "11",

journal = "Robotics",

issn = "2218-6581",

publisher = "M D P I AG",

number = "5",

}

TY - JOUR

T1 - Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots

AU - Ugurlu, Halil Ibrahim

AU - Pham, Xuan Huy

AU - Kayacan, Erdal

PY - 2022/10

Y1 - 2022/10

N2 - In this study, a novel end-to-end path planning algorithm based on deep reinforcement learning is proposed for aerial robots deployed in dense environments. The learning agent finds an obstacle-free way around the provided rough, global path by only depending on the observations from a forward-facing depth camera. A novel deep reinforcement learning framework is proposed to train the end-to-end policy with the capability of safely avoiding obstacles. The Webots open-source robot simulator is utilized for training the policy, introducing highly randomized environmental configurations for better generalization. The training is performed without dynamics calculations through randomized position updates to minimize the amount of data processed. The trained policy is first comprehensively evaluated in simulations involving physical dynamics and software-in-the-loop flight control. The proposed method is proven to have a 38% and 50% higher success rate compared to both deep reinforcement learning-based and artificial potential field-based baselines, respectively. The generalization capability of the method is verified in simulation-to-real transfer without further training. Real-time experiments are conducted with several trials in two different scenarios, showing a 50% higher success rate of the proposed method compared to the deep reinforcement learning-based baseline.

AB - In this study, a novel end-to-end path planning algorithm based on deep reinforcement learning is proposed for aerial robots deployed in dense environments. The learning agent finds an obstacle-free way around the provided rough, global path by only depending on the observations from a forward-facing depth camera. A novel deep reinforcement learning framework is proposed to train the end-to-end policy with the capability of safely avoiding obstacles. The Webots open-source robot simulator is utilized for training the policy, introducing highly randomized environmental configurations for better generalization. The training is performed without dynamics calculations through randomized position updates to minimize the amount of data processed. The trained policy is first comprehensively evaluated in simulations involving physical dynamics and software-in-the-loop flight control. The proposed method is proven to have a 38% and 50% higher success rate compared to both deep reinforcement learning-based and artificial potential field-based baselines, respectively. The generalization capability of the method is verified in simulation-to-real transfer without further training. Real-time experiments are conducted with several trials in two different scenarios, showing a 50% higher success rate of the proposed method compared to the deep reinforcement learning-based baseline.

KW - deep reinforcement learning

KW - obstacle avoidance

KW - quadrotors

KW - sim-to-real transfer

UR - http://www.scopus.com/inward/record.url?scp=85140897101&partnerID=8YFLogxK

U2 - 10.3390/robotics11050109

DO - 10.3390/robotics11050109

M3 - Journal article

SN - 2218-6581

VL - 11

JO - Robotics

JF - Robotics

IS - 5

M1 - 109

ER -

Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this