TY - JOUR

T1 - Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems

AU - Foufoula‐Georgiou, Efi

AU - Kitanidis, Peter K.

N1 - Copyright:
Copyright 2015 Elsevier B.V., All rights reserved.

PY - 1988/8

Y1 - 1988/8

N2 - A new computational algorithm is presented for the solution of discrete time linearly constrained stochastic optimal control problems decomposable in stages. The algorithm, designated gradient dynamic programming, is a backward moving stagewise optimization. The main innovations over conventional discrete dynamic programming (DDP) are in the functional representation of the cost‐to‐go function and the solution of the single‐stage problem. The cost‐to‐go function (assumed to be of requisite smoothness) is approximated within each element defined by the discretization scheme by the lowest‐order polynomial which preserve its values and the values of its gradient with respect to the state variables at all nodes of the discretization grid. The improved accuracy of this Hermitian interpolation scheme reduces the effect of discretization error and allows the use of coarser grids which reduces the dimensionality of the problem. At each stage, the optimal control is determined on each node of the discretized state space using a constrained Newton‐type optimization procedure which has quadratic rate of convergence. The set of constraints which act as equalities is determined from an active set strategy which converges under lenient convexity requirements. This method of solving the single‐stage optimization is much more efficient than the conventional way based on enumeration or iterative methods with linear rate of convergence. Once the optimal control is determined, the cost‐to‐go function and its gradient with respect to the state variables is calculated to be used at the next stage. The proposed technique permits the efficient optimization of stochastic systems whose high dimensionality does not permit solution under the conventional DDP framework and for which successive approximation methods are not directly applicable due to stochasticity. Results for a four‐reservoir example are presented.

AB - A new computational algorithm is presented for the solution of discrete time linearly constrained stochastic optimal control problems decomposable in stages. The algorithm, designated gradient dynamic programming, is a backward moving stagewise optimization. The main innovations over conventional discrete dynamic programming (DDP) are in the functional representation of the cost‐to‐go function and the solution of the single‐stage problem. The cost‐to‐go function (assumed to be of requisite smoothness) is approximated within each element defined by the discretization scheme by the lowest‐order polynomial which preserve its values and the values of its gradient with respect to the state variables at all nodes of the discretization grid. The improved accuracy of this Hermitian interpolation scheme reduces the effect of discretization error and allows the use of coarser grids which reduces the dimensionality of the problem. At each stage, the optimal control is determined on each node of the discretized state space using a constrained Newton‐type optimization procedure which has quadratic rate of convergence. The set of constraints which act as equalities is determined from an active set strategy which converges under lenient convexity requirements. This method of solving the single‐stage optimization is much more efficient than the conventional way based on enumeration or iterative methods with linear rate of convergence. Once the optimal control is determined, the cost‐to‐go function and its gradient with respect to the state variables is calculated to be used at the next stage. The proposed technique permits the efficient optimization of stochastic systems whose high dimensionality does not permit solution under the conventional DDP framework and for which successive approximation methods are not directly applicable due to stochasticity. Results for a four‐reservoir example are presented.

UR - http://www.scopus.com/inward/record.url?scp=0024158874&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0024158874&partnerID=8YFLogxK

U2 - 10.1029/WR024i008p01345

DO - 10.1029/WR024i008p01345

M3 - Article

AN - SCOPUS:0024158874

VL - 24

SP - 1345

EP - 1359

JO - Water Resources Research

JF - Water Resources Research

SN - 0043-1397

IS - 8

ER -