Global gridded crop model evaluation: Benchmarking, skills, deficiencies and implications

Christoph Müller, Joshua Elliott, James Chryssanthacopoulos, Almut Arneth, Juraj Balkovic, Philippe Ciais, Delphine Deryng, Christian Folberth, Michael Glotter, Steven Hoek, Toshichika Iizumi, Roberto C. Izaurralde, Curtis Jones, Nikolay Khabarov, Peter Lawrence, Wenfeng Liu, Stefan Olin, Thomas A.M. Pugh, Deepak K. Ray, Ashwan ReddyCynthia Rosenzweig, Alex C. Ruane, Gen Sakurai, Erwin Schmid, Rastislav Skalsky, Carol X. Song, Xuhui Wang, Allard De Wit, Hong Yang

Research output: Contribution to journalArticle

57 Citations (Scopus)

Abstract

Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.

Original languageEnglish (US)
Pages (from-to)1403-1422
Number of pages20
JournalGeoscientific Model Development
Volume10
Issue number4
DOIs
StatePublished - Apr 4 2017

Fingerprint

Model Evaluation
benchmarking
Benchmarking
Crops
crop
Soybean
Maize
Wheat
Performance Model
soybean
Time series
wheat
maize
Model
Benchmark
Modeling
rice
Simulation
Ensemble
Skills

Cite this

Müller, C., Elliott, J., Chryssanthacopoulos, J., Arneth, A., Balkovic, J., Ciais, P., ... Yang, H. (2017). Global gridded crop model evaluation: Benchmarking, skills, deficiencies and implications. Geoscientific Model Development, 10(4), 1403-1422. https://doi.org/10.5194/gmd-10-1403-2017

Global gridded crop model evaluation : Benchmarking, skills, deficiencies and implications. / Müller, Christoph; Elliott, Joshua; Chryssanthacopoulos, James; Arneth, Almut; Balkovic, Juraj; Ciais, Philippe; Deryng, Delphine; Folberth, Christian; Glotter, Michael; Hoek, Steven; Iizumi, Toshichika; Izaurralde, Roberto C.; Jones, Curtis; Khabarov, Nikolay; Lawrence, Peter; Liu, Wenfeng; Olin, Stefan; Pugh, Thomas A.M.; Ray, Deepak K.; Reddy, Ashwan; Rosenzweig, Cynthia; Ruane, Alex C.; Sakurai, Gen; Schmid, Erwin; Skalsky, Rastislav; Song, Carol X.; Wang, Xuhui; De Wit, Allard; Yang, Hong.

In: Geoscientific Model Development, Vol. 10, No. 4, 04.04.2017, p. 1403-1422.

Research output: Contribution to journalArticle

Müller, C, Elliott, J, Chryssanthacopoulos, J, Arneth, A, Balkovic, J, Ciais, P, Deryng, D, Folberth, C, Glotter, M, Hoek, S, Iizumi, T, Izaurralde, RC, Jones, C, Khabarov, N, Lawrence, P, Liu, W, Olin, S, Pugh, TAM, Ray, DK, Reddy, A, Rosenzweig, C, Ruane, AC, Sakurai, G, Schmid, E, Skalsky, R, Song, CX, Wang, X, De Wit, A & Yang, H 2017, 'Global gridded crop model evaluation: Benchmarking, skills, deficiencies and implications', Geoscientific Model Development, vol. 10, no. 4, pp. 1403-1422. https://doi.org/10.5194/gmd-10-1403-2017
Müller C, Elliott J, Chryssanthacopoulos J, Arneth A, Balkovic J, Ciais P et al. Global gridded crop model evaluation: Benchmarking, skills, deficiencies and implications. Geoscientific Model Development. 2017 Apr 4;10(4):1403-1422. https://doi.org/10.5194/gmd-10-1403-2017
Müller, Christoph ; Elliott, Joshua ; Chryssanthacopoulos, James ; Arneth, Almut ; Balkovic, Juraj ; Ciais, Philippe ; Deryng, Delphine ; Folberth, Christian ; Glotter, Michael ; Hoek, Steven ; Iizumi, Toshichika ; Izaurralde, Roberto C. ; Jones, Curtis ; Khabarov, Nikolay ; Lawrence, Peter ; Liu, Wenfeng ; Olin, Stefan ; Pugh, Thomas A.M. ; Ray, Deepak K. ; Reddy, Ashwan ; Rosenzweig, Cynthia ; Ruane, Alex C. ; Sakurai, Gen ; Schmid, Erwin ; Skalsky, Rastislav ; Song, Carol X. ; Wang, Xuhui ; De Wit, Allard ; Yang, Hong. / Global gridded crop model evaluation : Benchmarking, skills, deficiencies and implications. In: Geoscientific Model Development. 2017 ; Vol. 10, No. 4. pp. 1403-1422.
@article{673f5787738245c09ae1828f388930e0,
title = "Global gridded crop model evaluation: Benchmarking, skills, deficiencies and implications",
abstract = "Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.",
author = "Christoph M{\"u}ller and Joshua Elliott and James Chryssanthacopoulos and Almut Arneth and Juraj Balkovic and Philippe Ciais and Delphine Deryng and Christian Folberth and Michael Glotter and Steven Hoek and Toshichika Iizumi and Izaurralde, {Roberto C.} and Curtis Jones and Nikolay Khabarov and Peter Lawrence and Wenfeng Liu and Stefan Olin and Pugh, {Thomas A.M.} and Ray, {Deepak K.} and Ashwan Reddy and Cynthia Rosenzweig and Ruane, {Alex C.} and Gen Sakurai and Erwin Schmid and Rastislav Skalsky and Song, {Carol X.} and Xuhui Wang and {De Wit}, Allard and Hong Yang",
year = "2017",
month = "4",
day = "4",
doi = "10.5194/gmd-10-1403-2017",
language = "English (US)",
volume = "10",
pages = "1403--1422",
journal = "Geoscientific Model Development",
issn = "1991-959X",
publisher = "Copernicus Gesellschaft mbH",
number = "4",

}

TY - JOUR

T1 - Global gridded crop model evaluation

T2 - Benchmarking, skills, deficiencies and implications

AU - Müller, Christoph

AU - Elliott, Joshua

AU - Chryssanthacopoulos, James

AU - Arneth, Almut

AU - Balkovic, Juraj

AU - Ciais, Philippe

AU - Deryng, Delphine

AU - Folberth, Christian

AU - Glotter, Michael

AU - Hoek, Steven

AU - Iizumi, Toshichika

AU - Izaurralde, Roberto C.

AU - Jones, Curtis

AU - Khabarov, Nikolay

AU - Lawrence, Peter

AU - Liu, Wenfeng

AU - Olin, Stefan

AU - Pugh, Thomas A.M.

AU - Ray, Deepak K.

AU - Reddy, Ashwan

AU - Rosenzweig, Cynthia

AU - Ruane, Alex C.

AU - Sakurai, Gen

AU - Schmid, Erwin

AU - Skalsky, Rastislav

AU - Song, Carol X.

AU - Wang, Xuhui

AU - De Wit, Allard

AU - Yang, Hong

PY - 2017/4/4

Y1 - 2017/4/4

N2 - Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.

AB - Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.

UR - http://www.scopus.com/inward/record.url?scp=85017169636&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85017169636&partnerID=8YFLogxK

U2 - 10.5194/gmd-10-1403-2017

DO - 10.5194/gmd-10-1403-2017

M3 - Article

AN - SCOPUS:85017169636

VL - 10

SP - 1403

EP - 1422

JO - Geoscientific Model Development

JF - Geoscientific Model Development

SN - 1991-959X

IS - 4

ER -