Imperfect segmentation labels: How much do they matter?

Nicholas Heller, Joshua Dean, Nikolaos P Papanikolopoulos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Labeled datasets for semantic segmentation are imperfect, especially in medical imaging where borders are often subtle or ill-defined. Little work has been done to analyze the effect that label errors have on the performance of segmentation methodologies. Here we present a large-scale study of model performance in the presence of varying types and degrees of error in training data. We trained U-Net, SegNet, and FCN32 several times for liver segmentation with 10 different modes of ground-truth perturbation. Our results show that for each architecture, performance steadily declines with boundary-localized errors, however, U-Net was significantly more robust to jagged boundary errors than the other architectures. We also found that each architecture was very robust to non-boundary-localized errors, suggesting that boundary-localized errors are fundamentally different and more challenging problem than random label errors in a classification setting.

Original languageEnglish (US)
Title of host publicationIntravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018
EditorsSu-Lin Lee, Emanuele Trucco, Lena Maier-Hein, Stefano Moriconi, Shadi Albarqouni, Pierre Jannin, Simone Balocco, Guillaume Zahnd, Diana Mateus, Zeike Taylor, Stefanie Demirci, Danail Stoyanov, Raphael Sznitman, Anne Martel, Veronika Cheplygina, Eric Granger, Luc Duong
PublisherSpringer- Verlag
Pages112-120
Number of pages9
ISBN (Print)9783030013639
DOIs
StatePublished - Jan 1 2018
Event7th Joint International Workshop on Computing and Visualization for Intravascular Imaging and Computer Assisted Stenting, CVII-STENT 2018, and the 3rd International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2018, held in conjunction with the 21th International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018 - Granada, Spain
Duration: Sep 16 2018Sep 16 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11043 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other7th Joint International Workshop on Computing and Visualization for Intravascular Imaging and Computer Assisted Stenting, CVII-STENT 2018, and the 3rd International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2018, held in conjunction with the 21th International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018
CountrySpain
CityGranada
Period9/16/189/16/18

Fingerprint

Imperfect
Labels
Segmentation
Medical Imaging
Medical imaging
Performance Model
Liver
Semantics
Perturbation
Methodology
Architecture

Cite this

Heller, N., Dean, J., & Papanikolopoulos, N. P. (2018). Imperfect segmentation labels: How much do they matter? In S-L. Lee, E. Trucco, L. Maier-Hein, S. Moriconi, S. Albarqouni, P. Jannin, S. Balocco, G. Zahnd, D. Mateus, Z. Taylor, S. Demirci, D. Stoyanov, R. Sznitman, A. Martel, V. Cheplygina, E. Granger, ... L. Duong (Eds.), Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018 (pp. 112-120). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11043 LNCS). Springer- Verlag. https://doi.org/10.1007/978-3-030-01364-6_13

Imperfect segmentation labels : How much do they matter? / Heller, Nicholas; Dean, Joshua; Papanikolopoulos, Nikolaos P.

Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018. ed. / Su-Lin Lee; Emanuele Trucco; Lena Maier-Hein; Stefano Moriconi; Shadi Albarqouni; Pierre Jannin; Simone Balocco; Guillaume Zahnd; Diana Mateus; Zeike Taylor; Stefanie Demirci; Danail Stoyanov; Raphael Sznitman; Anne Martel; Veronika Cheplygina; Eric Granger; Luc Duong. Springer- Verlag, 2018. p. 112-120 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11043 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Heller, N, Dean, J & Papanikolopoulos, NP 2018, Imperfect segmentation labels: How much do they matter? in S-L Lee, E Trucco, L Maier-Hein, S Moriconi, S Albarqouni, P Jannin, S Balocco, G Zahnd, D Mateus, Z Taylor, S Demirci, D Stoyanov, R Sznitman, A Martel, V Cheplygina, E Granger & L Duong (eds), Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11043 LNCS, Springer- Verlag, pp. 112-120, 7th Joint International Workshop on Computing and Visualization for Intravascular Imaging and Computer Assisted Stenting, CVII-STENT 2018, and the 3rd International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2018, held in conjunction with the 21th International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018, Granada, Spain, 9/16/18. https://doi.org/10.1007/978-3-030-01364-6_13
Heller N, Dean J, Papanikolopoulos NP. Imperfect segmentation labels: How much do they matter? In Lee S-L, Trucco E, Maier-Hein L, Moriconi S, Albarqouni S, Jannin P, Balocco S, Zahnd G, Mateus D, Taylor Z, Demirci S, Stoyanov D, Sznitman R, Martel A, Cheplygina V, Granger E, Duong L, editors, Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018. Springer- Verlag. 2018. p. 112-120. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01364-6_13
Heller, Nicholas ; Dean, Joshua ; Papanikolopoulos, Nikolaos P. / Imperfect segmentation labels : How much do they matter?. Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018. editor / Su-Lin Lee ; Emanuele Trucco ; Lena Maier-Hein ; Stefano Moriconi ; Shadi Albarqouni ; Pierre Jannin ; Simone Balocco ; Guillaume Zahnd ; Diana Mateus ; Zeike Taylor ; Stefanie Demirci ; Danail Stoyanov ; Raphael Sznitman ; Anne Martel ; Veronika Cheplygina ; Eric Granger ; Luc Duong. Springer- Verlag, 2018. pp. 112-120 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{589880d085cf49e5a257ed53b047e648,
title = "Imperfect segmentation labels: How much do they matter?",
abstract = "Labeled datasets for semantic segmentation are imperfect, especially in medical imaging where borders are often subtle or ill-defined. Little work has been done to analyze the effect that label errors have on the performance of segmentation methodologies. Here we present a large-scale study of model performance in the presence of varying types and degrees of error in training data. We trained U-Net, SegNet, and FCN32 several times for liver segmentation with 10 different modes of ground-truth perturbation. Our results show that for each architecture, performance steadily declines with boundary-localized errors, however, U-Net was significantly more robust to jagged boundary errors than the other architectures. We also found that each architecture was very robust to non-boundary-localized errors, suggesting that boundary-localized errors are fundamentally different and more challenging problem than random label errors in a classification setting.",
author = "Nicholas Heller and Joshua Dean and Papanikolopoulos, {Nikolaos P}",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-01364-6_13",
language = "English (US)",
isbn = "9783030013639",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer- Verlag",
pages = "112--120",
editor = "Su-Lin Lee and Emanuele Trucco and Lena Maier-Hein and Stefano Moriconi and Shadi Albarqouni and Pierre Jannin and Simone Balocco and Guillaume Zahnd and Diana Mateus and Zeike Taylor and Stefanie Demirci and Danail Stoyanov and Raphael Sznitman and Anne Martel and Veronika Cheplygina and Eric Granger and Luc Duong",
booktitle = "Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018",

}

TY - GEN

T1 - Imperfect segmentation labels

T2 - How much do they matter?

AU - Heller, Nicholas

AU - Dean, Joshua

AU - Papanikolopoulos, Nikolaos P

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Labeled datasets for semantic segmentation are imperfect, especially in medical imaging where borders are often subtle or ill-defined. Little work has been done to analyze the effect that label errors have on the performance of segmentation methodologies. Here we present a large-scale study of model performance in the presence of varying types and degrees of error in training data. We trained U-Net, SegNet, and FCN32 several times for liver segmentation with 10 different modes of ground-truth perturbation. Our results show that for each architecture, performance steadily declines with boundary-localized errors, however, U-Net was significantly more robust to jagged boundary errors than the other architectures. We also found that each architecture was very robust to non-boundary-localized errors, suggesting that boundary-localized errors are fundamentally different and more challenging problem than random label errors in a classification setting.

AB - Labeled datasets for semantic segmentation are imperfect, especially in medical imaging where borders are often subtle or ill-defined. Little work has been done to analyze the effect that label errors have on the performance of segmentation methodologies. Here we present a large-scale study of model performance in the presence of varying types and degrees of error in training data. We trained U-Net, SegNet, and FCN32 several times for liver segmentation with 10 different modes of ground-truth perturbation. Our results show that for each architecture, performance steadily declines with boundary-localized errors, however, U-Net was significantly more robust to jagged boundary errors than the other architectures. We also found that each architecture was very robust to non-boundary-localized errors, suggesting that boundary-localized errors are fundamentally different and more challenging problem than random label errors in a classification setting.

UR - http://www.scopus.com/inward/record.url?scp=85055816105&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055816105&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01364-6_13

DO - 10.1007/978-3-030-01364-6_13

M3 - Conference contribution

SN - 9783030013639

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 112

EP - 120

BT - Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018

A2 - Lee, Su-Lin

A2 - Trucco, Emanuele

A2 - Maier-Hein, Lena

A2 - Moriconi, Stefano

A2 - Albarqouni, Shadi

A2 - Jannin, Pierre

A2 - Balocco, Simone

A2 - Zahnd, Guillaume

A2 - Mateus, Diana

A2 - Taylor, Zeike

A2 - Demirci, Stefanie

A2 - Stoyanov, Danail

A2 - Sznitman, Raphael

A2 - Martel, Anne

A2 - Cheplygina, Veronika

A2 - Granger, Eric

A2 - Duong, Luc

PB - Springer- Verlag

ER -