Recovering robustness in model-free reinforcement learning

Harish K. Venkataraman; Peter J. Seiler

doi:10.23919/acc.2019.8815368

Recovering robustness in model-free reinforcement learning

Harish K. Venkataraman, Peter J. Seiler

Aerospace Engineering and Mechanics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

11 Scopus citations

Abstract

Reinforcement learning (RL) is used to directly design a control policy using data collected from the system. This paper considers the robustness of controllers trained via model-free RL. The discussion focuses on posing the (model-free) linear quadratic Gaussian (LQG) problem as a special instance of RL. A simple LQG example is used to demonstrate that RL with partial observations can lead to poor robustness margins. It is proposed to recover robustness by introducing random perturbations at the system input during the RL training. The perturbation magnitude can be used to trade off performance for increased robustness. Two simple examples are presented to demonstrate the proposed method for enhancing robustness during RL training.

Original language	English (US)
Title of host publication	2019 American Control Conference, ACC 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	4210-4216
Number of pages	7
ISBN (Electronic)	9781538679265
DOIs	https://doi.org/10.23919/acc.2019.8815368
State	Published - Jul 2019
Event	2019 American Control Conference, ACC 2019 - Philadelphia, United States Duration: Jul 10 2019 → Jul 12 2019

Publication series

Name	Proceedings of the American Control Conference
Volume	2019-July
ISSN (Print)	0743-1619

Conference

Conference	2019 American Control Conference, ACC 2019
Country/Territory	United States
City	Philadelphia
Period	7/10/19 → 7/12/19

Bibliographical note

Publisher Copyright:
© 2019 American Automatic Control Council.

Publisher link

10.23919/acc.2019.8815368

Find It

Find It at U of M Libraries

Cite this

Recovering robustness in model-free reinforcement learning. / Venkataraman, Harish K.; Seiler, Peter J.
2019 American Control Conference, ACC 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 4210-4216 8815368 (Proceedings of the American Control Conference; Vol. 2019-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Venkataraman, HK & Seiler, PJ 2019, Recovering robustness in model-free reinforcement learning. in 2019 American Control Conference, ACC 2019., 8815368, Proceedings of the American Control Conference, vol. 2019-July, Institute of Electrical and Electronics Engineers Inc., pp. 4210-4216, 2019 American Control Conference, ACC 2019, Philadelphia, United States, 7/10/19. https://doi.org/10.23919/acc.2019.8815368

@inproceedings{0779d630846542fb89f111316dec061e,

title = "Recovering robustness in model-free reinforcement learning",

abstract = "Reinforcement learning (RL) is used to directly design a control policy using data collected from the system. This paper considers the robustness of controllers trained via model-free RL. The discussion focuses on posing the (model-free) linear quadratic Gaussian (LQG) problem as a special instance of RL. A simple LQG example is used to demonstrate that RL with partial observations can lead to poor robustness margins. It is proposed to recover robustness by introducing random perturbations at the system input during the RL training. The perturbation magnitude can be used to trade off performance for increased robustness. Two simple examples are presented to demonstrate the proposed method for enhancing robustness during RL training.",

author = "Venkataraman, {Harish K.} and Seiler, {Peter J.}",

note = "Publisher Copyright: {\textcopyright} 2019 American Automatic Control Council.; 2019 American Control Conference, ACC 2019 ; Conference date: 10-07-2019 Through 12-07-2019",

year = "2019",

month = jul,

doi = "10.23919/acc.2019.8815368",

language = "English (US)",

series = "Proceedings of the American Control Conference",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "4210--4216",

booktitle = "2019 American Control Conference, ACC 2019",

}

TY - GEN

T1 - Recovering robustness in model-free reinforcement learning

AU - Venkataraman, Harish K.

AU - Seiler, Peter J.

PY - 2019/7

Y1 - 2019/7

N2 - Reinforcement learning (RL) is used to directly design a control policy using data collected from the system. This paper considers the robustness of controllers trained via model-free RL. The discussion focuses on posing the (model-free) linear quadratic Gaussian (LQG) problem as a special instance of RL. A simple LQG example is used to demonstrate that RL with partial observations can lead to poor robustness margins. It is proposed to recover robustness by introducing random perturbations at the system input during the RL training. The perturbation magnitude can be used to trade off performance for increased robustness. Two simple examples are presented to demonstrate the proposed method for enhancing robustness during RL training.

AB - Reinforcement learning (RL) is used to directly design a control policy using data collected from the system. This paper considers the robustness of controllers trained via model-free RL. The discussion focuses on posing the (model-free) linear quadratic Gaussian (LQG) problem as a special instance of RL. A simple LQG example is used to demonstrate that RL with partial observations can lead to poor robustness margins. It is proposed to recover robustness by introducing random perturbations at the system input during the RL training. The perturbation magnitude can be used to trade off performance for increased robustness. Two simple examples are presented to demonstrate the proposed method for enhancing robustness during RL training.

UR - http://www.scopus.com/inward/record.url?scp=85072295039&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072295039&partnerID=8YFLogxK

U2 - 10.23919/acc.2019.8815368

DO - 10.23919/acc.2019.8815368

M3 - Conference contribution

AN - SCOPUS:85072295039

T3 - Proceedings of the American Control Conference

SP - 4210

EP - 4216

BT - 2019 American Control Conference, ACC 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2019 American Control Conference, ACC 2019

Y2 - 10 July 2019 through 12 July 2019

ER -

Recovering robustness in model-free reinforcement learning

Abstract

Publication series

Conference

Bibliographical note

Publisher link

Find It

Other files and links

Fingerprint

Cite this