Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software

Daniel Sage; Thanh An Pham; Hazen Babcock; Tomas Lukes; Thomas Pengo; Jerry Chao; Ramraj Velmurugan; Alex Herbert; Anurag Agrawal; Silvia Colabrese; Ann Wheeler; Anna Archetti; Bernd Rieger; Raimund Ober; Guy M. Hagen; Jean Baptiste Sibarita; Jonas Ries; Ricardo Henriques; Michael Unser; Seamus Holden

doi:10.1038/s41592-019-0364-4

Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software

Daniel Sage, Thanh An Pham, Hazen Babcock, Tomas Lukes, Thomas Pengo, Jerry Chao, Ramraj Velmurugan, Alex Herbert, Anurag Agrawal, Silvia Colabrese, Ann Wheeler, Anna Archetti, Bernd Rieger, Raimund Ober, Guy M. Hagen, Jean Baptiste Sibarita, Jonas Ries, Ricardo Henriques, Michael Unser, Seamus Holden

University of Minnesota Informatics Institute

Research output: Contribution to journal › Article › peer-review

237 Scopus citations

Abstract

With the widespread uptake of two-dimensional (2D) and three-dimensional (3D) single-molecule localization microscopy (SMLM), a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort, we designed a competition to extensively characterize and rank the performance of 2D and 3D SMLM software packages. We generated realistic simulated datasets for popular imaging modalities—2D, astigmatic 3D, biplane 3D and double-helix 3D—and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D SMLM software and provides a holistic view of how the latest 2D and 3D SMLM packages perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against the current state of the art, and provides insight into the current limits of the field.

Original language	English (US)
Pages (from-to)	387-395
Number of pages	9
Journal	Nature Methods
Volume	16
Issue number	5
DOIs	https://doi.org/10.1038/s41592-019-0364-4
State	Published - May 1 2019

Bibliographical note

Funding Information:
The authors acknowledge the following funding sources: a Newcastle University Research Fellowship and a Wellcome Trust and Royal Society Sir Henry Dale Fellowship grant number 206670/Z/17/Z to S.H.; a European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program, Grant Agreement number 692726 to D.S., T.A.P., and M.U.; UK BBSRC grants BB/M022374/1, BB/ P027431/1, BB/R000697/1 grant and MRC grants MC-UU-12018/2, MR/K015826/1 to R.H.; ERC grant CoG-724489, CellStructure to J.R.; FranceBioImaging infrastructure ANR-10-INBS-04 to J.-B.S.; National Institutes of Health grant 1R15GM128166-01 to G.M.H.; and NSF SBIR grants 1353638, 1534745 to Double Helix LLC. We thank R. Piestun at University of Colorado for providing double-helix PSF phase mask designs to Double Helix LLC. We thank all the localization microscopy challenge participants for their contribution: H. Babcock (3D-DAOSTORM, Cspline, L1H); F. Hauser (3D STORM Tools); S. Watanabe (3D-WTM,WTM); N. Boyd (ADCG); J. Min, K. Jin and J.C. Ye (ALOHA, FALCON); H. Rouault (B-recs); E. Soubies (CEL0-STORM); A. Speiser, S. Turagas and J. Macke (DECODE); A. von Diezmann, C. Bayas and W.E. Moerner (Easy-DHPSF); T. Vomhof and J. Reichel (FIRESTORM); H. Pan (LEAP); A. Wheeler (Localizer); Z.-l. Huang and Y. Wang (MaLiang); J. Chao, R. Velmurugan, A.V. Abraham and R.J. Ober (MIATool); H. Deschout (mlePALM); T. Pengo (Octane, PeakSelector); Y.-n. Wang (PALMER); A. Herbert (PeakFit); K. Martens and J. Hohlbein (pSMLM-3D); L. Li (QC-STORM); R. Henriques (QuickPALM); G. Tamas and J. Sinko (RainSTORM); S. Wolter and M. Sauer (RapidSTORM); M. Kirchgessner and F. Gruell (SFP Estimator); Y. Li and J. Ries (SMAP); H. Ikoma (SMfit); A. Loot, A. Valdmann, M. Eltermann, M. Kree and M. Pärs (SMolPhot); Y.J. Jung, A. Barsic, R. Piestun and N. Fakhri (SOLAR_STORM); A. Archetti (STORMChaser); M. Ovesny, G. Hagen and P. Krizek (ThunderSTORM); J. Huang (TVSTORM); A. Kechkar, C. Butler and J.-B. Sibarita (WaveTracer) and B. Lelandais (ZOLA-3D). We thank the SMLMS 2016 organizers (S. Manley and A. Radenovic, EPFL) for hosting a localization microscopy challenge special session. We also thank Double Helix and Molecular Devices for sponsoring the SMLMS 2016 special session. The sponsors had no input or influence on the research.

Publisher Copyright:
© 2019, The Author(s), under exclusive licence to Springer Nature America, Inc.

PubMed: MeSH publication types

Journal Article
Research Support, Non-U.S. Gov't

Publisher link

10.1038/s41592-019-0364-4

https://eprints.soton.ac.uk/430424/1/2018_SMLM_Challenge_16_accepted_version_final.pdf

Find It

Find It at U of M Libraries

Cite this

Sage, D., Pham, T. A., Babcock, H., Lukes, T., Pengo, T., Chao, J., Velmurugan, R., Herbert, A., Agrawal, A., Colabrese, S., Wheeler, A., Archetti, A., Rieger, B., Ober, R., Hagen, G. M., Sibarita, J. B., Ries, J., Henriques, R., Unser, M., & Holden, S. (2019). Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software. Nature Methods, 16(5), 387-395. https://doi.org/10.1038/s41592-019-0364-4

Sage, D, Pham, TA, Babcock, H, Lukes, T, Pengo, T, Chao, J, Velmurugan, R, Herbert, A, Agrawal, A, Colabrese, S, Wheeler, A, Archetti, A, Rieger, B, Ober, R, Hagen, GM, Sibarita, JB, Ries, J, Henriques, R, Unser, M & Holden, S 2019, 'Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software', Nature Methods, vol. 16, no. 5, pp. 387-395. https://doi.org/10.1038/s41592-019-0364-4

@article{70eebe0200c94452afcad0f7541c382d,

title = "Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software",

abstract = "With the widespread uptake of two-dimensional (2D) and three-dimensional (3D) single-molecule localization microscopy (SMLM), a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort, we designed a competition to extensively characterize and rank the performance of 2D and 3D SMLM software packages. We generated realistic simulated datasets for popular imaging modalities—2D, astigmatic 3D, biplane 3D and double-helix 3D—and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D SMLM software and provides a holistic view of how the latest 2D and 3D SMLM packages perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against the current state of the art, and provides insight into the current limits of the field.",

author = "Daniel Sage and Pham, {Thanh An} and Hazen Babcock and Tomas Lukes and Thomas Pengo and Jerry Chao and Ramraj Velmurugan and Alex Herbert and Anurag Agrawal and Silvia Colabrese and Ann Wheeler and Anna Archetti and Bernd Rieger and Raimund Ober and Hagen, {Guy M.} and Sibarita, {Jean Baptiste} and Jonas Ries and Ricardo Henriques and Michael Unser and Seamus Holden",

note = "Funding Information: The authors acknowledge the following funding sources: a Newcastle University Research Fellowship and a Wellcome Trust and Royal Society Sir Henry Dale Fellowship grant number 206670/Z/17/Z to S.H.; a European Research Council (ERC) under the European Union{\textquoteright}s Horizon 2020 research and innovation program, Grant Agreement number 692726 to D.S., T.A.P., and M.U.; UK BBSRC grants BB/M022374/1, BB/ P027431/1, BB/R000697/1 grant and MRC grants MC-UU-12018/2, MR/K015826/1 to R.H.; ERC grant CoG-724489, CellStructure to J.R.; FranceBioImaging infrastructure ANR-10-INBS-04 to J.-B.S.; National Institutes of Health grant 1R15GM128166-01 to G.M.H.; and NSF SBIR grants 1353638, 1534745 to Double Helix LLC. We thank R. Piestun at University of Colorado for providing double-helix PSF phase mask designs to Double Helix LLC. We thank all the localization microscopy challenge participants for their contribution: H. Babcock (3D-DAOSTORM, Cspline, L1H); F. Hauser (3D STORM Tools); S. Watanabe (3D-WTM,WTM); N. Boyd (ADCG); J. Min, K. Jin and J.C. Ye (ALOHA, FALCON); H. Rouault (B-recs); E. Soubies (CEL0-STORM); A. Speiser, S. Turagas and J. Macke (DECODE); A. von Diezmann, C. Bayas and W.E. Moerner (Easy-DHPSF); T. Vomhof and J. Reichel (FIRESTORM); H. Pan (LEAP); A. Wheeler (Localizer); Z.-l. Huang and Y. Wang (MaLiang); J. Chao, R. Velmurugan, A.V. Abraham and R.J. Ober (MIATool); H. Deschout (mlePALM); T. Pengo (Octane, PeakSelector); Y.-n. Wang (PALMER); A. Herbert (PeakFit); K. Martens and J. Hohlbein (pSMLM-3D); L. Li (QC-STORM); R. Henriques (QuickPALM); G. Tamas and J. Sinko (RainSTORM); S. Wolter and M. Sauer (RapidSTORM); M. Kirchgessner and F. Gruell (SFP Estimator); Y. Li and J. Ries (SMAP); H. Ikoma (SMfit); A. Loot, A. Valdmann, M. Eltermann, M. Kree and M. P{\"a}rs (SMolPhot); Y.J. Jung, A. Barsic, R. Piestun and N. Fakhri (SOLAR_STORM); A. Archetti (STORMChaser); M. Ovesny, G. Hagen and P. Krizek (ThunderSTORM); J. Huang (TVSTORM); A. Kechkar, C. Butler and J.-B. Sibarita (WaveTracer) and B. Lelandais (ZOLA-3D). We thank the SMLMS 2016 organizers (S. Manley and A. Radenovic, EPFL) for hosting a localization microscopy challenge special session. We also thank Double Helix and Molecular Devices for sponsoring the SMLMS 2016 special session. The sponsors had no input or influence on the research. Publisher Copyright: {\textcopyright} 2019, The Author(s), under exclusive licence to Springer Nature America, Inc.",

year = "2019",

month = may,

day = "1",

doi = "10.1038/s41592-019-0364-4",

language = "English (US)",

volume = "16",

pages = "387--395",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Public Library of Science",

number = "5",

}

TY - JOUR

T1 - Super-resolution fight club

T2 - assessment of 2D and 3D single-molecule localization microscopy software

AU - Sage, Daniel

AU - Pham, Thanh An

AU - Babcock, Hazen

AU - Lukes, Tomas

AU - Pengo, Thomas

AU - Chao, Jerry

AU - Velmurugan, Ramraj

AU - Herbert, Alex

AU - Agrawal, Anurag

AU - Colabrese, Silvia

AU - Wheeler, Ann

AU - Archetti, Anna

AU - Rieger, Bernd

AU - Ober, Raimund

AU - Hagen, Guy M.

AU - Sibarita, Jean Baptiste

AU - Ries, Jonas

AU - Henriques, Ricardo

AU - Unser, Michael

AU - Holden, Seamus

N1 - Funding Information: The authors acknowledge the following funding sources: a Newcastle University Research Fellowship and a Wellcome Trust and Royal Society Sir Henry Dale Fellowship grant number 206670/Z/17/Z to S.H.; a European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program, Grant Agreement number 692726 to D.S., T.A.P., and M.U.; UK BBSRC grants BB/M022374/1, BB/ P027431/1, BB/R000697/1 grant and MRC grants MC-UU-12018/2, MR/K015826/1 to R.H.; ERC grant CoG-724489, CellStructure to J.R.; FranceBioImaging infrastructure ANR-10-INBS-04 to J.-B.S.; National Institutes of Health grant 1R15GM128166-01 to G.M.H.; and NSF SBIR grants 1353638, 1534745 to Double Helix LLC. We thank R. Piestun at University of Colorado for providing double-helix PSF phase mask designs to Double Helix LLC. We thank all the localization microscopy challenge participants for their contribution: H. Babcock (3D-DAOSTORM, Cspline, L1H); F. Hauser (3D STORM Tools); S. Watanabe (3D-WTM,WTM); N. Boyd (ADCG); J. Min, K. Jin and J.C. Ye (ALOHA, FALCON); H. Rouault (B-recs); E. Soubies (CEL0-STORM); A. Speiser, S. Turagas and J. Macke (DECODE); A. von Diezmann, C. Bayas and W.E. Moerner (Easy-DHPSF); T. Vomhof and J. Reichel (FIRESTORM); H. Pan (LEAP); A. Wheeler (Localizer); Z.-l. Huang and Y. Wang (MaLiang); J. Chao, R. Velmurugan, A.V. Abraham and R.J. Ober (MIATool); H. Deschout (mlePALM); T. Pengo (Octane, PeakSelector); Y.-n. Wang (PALMER); A. Herbert (PeakFit); K. Martens and J. Hohlbein (pSMLM-3D); L. Li (QC-STORM); R. Henriques (QuickPALM); G. Tamas and J. Sinko (RainSTORM); S. Wolter and M. Sauer (RapidSTORM); M. Kirchgessner and F. Gruell (SFP Estimator); Y. Li and J. Ries (SMAP); H. Ikoma (SMfit); A. Loot, A. Valdmann, M. Eltermann, M. Kree and M. Pärs (SMolPhot); Y.J. Jung, A. Barsic, R. Piestun and N. Fakhri (SOLAR_STORM); A. Archetti (STORMChaser); M. Ovesny, G. Hagen and P. Krizek (ThunderSTORM); J. Huang (TVSTORM); A. Kechkar, C. Butler and J.-B. Sibarita (WaveTracer) and B. Lelandais (ZOLA-3D). We thank the SMLMS 2016 organizers (S. Manley and A. Radenovic, EPFL) for hosting a localization microscopy challenge special session. We also thank Double Helix and Molecular Devices for sponsoring the SMLMS 2016 special session. The sponsors had no input or influence on the research. Publisher Copyright: © 2019, The Author(s), under exclusive licence to Springer Nature America, Inc.

PY - 2019/5/1

Y1 - 2019/5/1

N2 - With the widespread uptake of two-dimensional (2D) and three-dimensional (3D) single-molecule localization microscopy (SMLM), a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort, we designed a competition to extensively characterize and rank the performance of 2D and 3D SMLM software packages. We generated realistic simulated datasets for popular imaging modalities—2D, astigmatic 3D, biplane 3D and double-helix 3D—and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D SMLM software and provides a holistic view of how the latest 2D and 3D SMLM packages perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against the current state of the art, and provides insight into the current limits of the field.

AB - With the widespread uptake of two-dimensional (2D) and three-dimensional (3D) single-molecule localization microscopy (SMLM), a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort, we designed a competition to extensively characterize and rank the performance of 2D and 3D SMLM software packages. We generated realistic simulated datasets for popular imaging modalities—2D, astigmatic 3D, biplane 3D and double-helix 3D—and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D SMLM software and provides a holistic view of how the latest 2D and 3D SMLM packages perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against the current state of the art, and provides insight into the current limits of the field.

UR - http://www.scopus.com/inward/record.url?scp=85064068699&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064068699&partnerID=8YFLogxK

U2 - 10.1038/s41592-019-0364-4

DO - 10.1038/s41592-019-0364-4

M3 - Article

C2 - 30962624

AN - SCOPUS:85064068699

SN - 1548-7091

VL - 16

SP - 387

EP - 395

JO - Nature Methods

JF - Nature Methods

IS - 5

ER -

Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software

Abstract

Bibliographical note

PubMed: MeSH publication types

Publisher link

Find It

Other files and links

Fingerprint

Cite this