Two-timescale voltage regulation in distribution grids using deep reinforcement learning

Frequent and sizeable voltage fluctuations become more pronounced with the increasing penetration of distributed renewable generation, and they considerably challenge distribution grids. Voltage regulation schemes so far have relied on either utility-owned devices (e.g., voltage transformers, and shunt capacitors), or more recently, smart power inverters that come with contemporary distributed generation units (e.g., photovoltaic systems, and wind turbines). Nonetheless, due to the distinct response times of those devices, as well as the discrete on-off commitment of capacitor units, joint control of both types of assets is challenging. In this context, a novel two-timescale voltage regulation scheme is developed here by coupling optimization with reinforcement learning advances. Shunt capacitors are configured on a slow timescale (e.g., daily basis) leveraging a deep reinforcement learning algorithm, while optimal setpoints of the power inverters are computed using a linearized distribution flow model on a fast timescale (e.g., every few seconds or minutes). Numerical experiments using a real-world 47-bus distribution feeder showcase the remarkable performance of the novel scheme.