Uncertainty Estimation with Distributional Reinforcement Learning for Applications in Intelligent Transportation Systems: A Case Study

Pengyue Wang, Yan Li, Shashi Shekhar, William F. Northrop

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Reinforcement learning (RL) algorithms have been successfully used in the area of Intelligent Transportation Systems (ITS) for applications such as energy management strategies (EMS) of hybrid electric vehicles, autonomous driving, traffic light cycle control and bottleneck decongestion. In this work, we investigate a distributional RL algorithm on an EMS problem as a case study to show the benefits of estimating the uncertainty associated with different actions at different states. The uncertainty estimation is highly beneficial to ITS applications as randomness and uncertainty are intrinsic characteristics of real-world problems due to incomplete knowledge of the environment and the stochastic nature of some real-world systems and human behaviors. The modeled uncertainty has the form of a return distribution for taking an action at a certain state. We provide a case study to show that only considering the expected reward value would result in a loss of important information such as the spread. By modeling the return distribution, domain knowledge can be incorporated to make the results more interpretable. Also, risk-sensitive strategies can be developed to build more robust solutions using a chosen utility function.

Original languageEnglish (US)
Title of host publication2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3822-3827
Number of pages6
ISBN (Electronic)9781538670248
DOIs
StatePublished - Oct 2019
Event2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019 - Auckland, New Zealand
Duration: Oct 27 2019Oct 30 2019

Publication series

Name2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019

Conference

Conference2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019
CountryNew Zealand
CityAuckland
Period10/27/1910/30/19

    Fingerprint

Cite this

Wang, P., Li, Y., Shekhar, S., & Northrop, W. F. (2019). Uncertainty Estimation with Distributional Reinforcement Learning for Applications in Intelligent Transportation Systems: A Case Study. In 2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019 (pp. 3822-3827). [8917429] (2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ITSC.2019.8917429