Large-Scale Analysis of the Docker Hub Dataset

Nannan Zhao; Vasily Tarasov; Hadeel Albahar; Ali Anwar; Lukas Rupprecht; Dimitrios Skourtis; Amit S. Warke; Mohamed Mohamed; Ali R. Butt

doi:10.1109/CLUSTER.2019.8891000

Large-Scale Analysis of the Docker Hub Dataset

Nannan Zhao, Vasily Tarasov, Hadeel Albahar, Ali Anwar, Lukas Rupprecht, Dimitrios Skourtis, Amit S. Warke, Mohamed Mohamed, Ali R. Butt

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

30 Scopus citations

Abstract

Docker containers have become a prominent solution for supporting modern enterprise applications due to the highly desirable features of isolation, low overhead, and efficient packaging of the execution environment. Containers are created from images which are shared between users via a Docker registry. The amount of data Docker registries store is massive; for example, Docker Hub, a popular public registry, stores at least half a million public images. In this paper, we analyze over 167 TB of uncompressed Docker Hub images, characterize them using multiple metrics and evaluate the potential of file-level deduplication in Docker Hub. Our analysis helps to make conscious decisions when designing storage for containers in general and Docker registries in particular. For example, only 3% of the files in images are unique, which means file-level deduplication has a great potential to save storage space for the registry. Our findings can motivate and help improve the design of data reduction, caching, and pulling optimizations for registries.

Original language	English (US)
Title of host publication	Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728147345
DOIs	https://doi.org/10.1109/CLUSTER.2019.8891000
State	Published - Sep 2019
Externally published	Yes
Event	2019 IEEE International Conference on Cluster Computing, CLUSTER 2019 - Albuquerque, United States Duration: Sep 23 2019 → Sep 26 2019

Publication series

Name	Proceedings - IEEE International Conference on Cluster Computing, ICCC
Volume	2019-September
ISSN (Print)	1552-5244

Conference

Conference	2019 IEEE International Conference on Cluster Computing, CLUSTER 2019
Country/Territory	United States
City	Albuquerque
Period	9/23/19 → 9/26/19

Bibliographical note

Funding Information:
Acknowledgments This work is sponsored by the NSF under the grants: CNS-1405697, CNS-1615411, and CNS-1565314/1838271.

Publisher Copyright:
© 2019 IEEE.

Publisher link

10.1109/CLUSTER.2019.8891000

Find It

Find It at U of M Libraries

Cite this

Zhao, N., Tarasov, V., Albahar, H., Anwar, A., Rupprecht, L., Skourtis, D., Warke, A. S., Mohamed, M., & Butt, A. R. (2019). Large-Scale Analysis of the Docker Hub Dataset. In Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019 Article 8891000 (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2019-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTER.2019.8891000

Large-Scale Analysis of the Docker Hub Dataset. / Zhao, Nannan; Tarasov, Vasily; Albahar, Hadeel et al.
Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019. Institute of Electrical and Electronics Engineers Inc., 2019. 8891000 (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2019-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Zhao, N, Tarasov, V, Albahar, H, Anwar, A, Rupprecht, L, Skourtis, D, Warke, AS, Mohamed, M & Butt, AR 2019, Large-Scale Analysis of the Docker Hub Dataset. in Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019., 8891000, Proceedings - IEEE International Conference on Cluster Computing, ICCC, vol. 2019-September, Institute of Electrical and Electronics Engineers Inc., 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019, Albuquerque, United States, 9/23/19. https://doi.org/10.1109/CLUSTER.2019.8891000

Zhao N, Tarasov V, Albahar H, Anwar A, Rupprecht L, Skourtis D et al. Large-Scale Analysis of the Docker Hub Dataset. In Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 8891000. (Proceedings - IEEE International Conference on Cluster Computing, ICCC). doi: 10.1109/CLUSTER.2019.8891000

@inproceedings{0b740784927748f0ac665b43f39126ad,

title = "Large-Scale Analysis of the Docker Hub Dataset",

abstract = "Docker containers have become a prominent solution for supporting modern enterprise applications due to the highly desirable features of isolation, low overhead, and efficient packaging of the execution environment. Containers are created from images which are shared between users via a Docker registry. The amount of data Docker registries store is massive; for example, Docker Hub, a popular public registry, stores at least half a million public images. In this paper, we analyze over 167 TB of uncompressed Docker Hub images, characterize them using multiple metrics and evaluate the potential of file-level deduplication in Docker Hub. Our analysis helps to make conscious decisions when designing storage for containers in general and Docker registries in particular. For example, only 3% of the files in images are unique, which means file-level deduplication has a great potential to save storage space for the registry. Our findings can motivate and help improve the design of data reduction, caching, and pulling optimizations for registries.",

author = "Nannan Zhao and Vasily Tarasov and Hadeel Albahar and Ali Anwar and Lukas Rupprecht and Dimitrios Skourtis and Warke, {Amit S.} and Mohamed Mohamed and Butt, {Ali R.}",

note = "Funding Information: Acknowledgments This work is sponsored by the NSF under the grants: CNS-1405697, CNS-1615411, and CNS-1565314/1838271. Publisher Copyright: {\textcopyright} 2019 IEEE.; 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019 ; Conference date: 23-09-2019 Through 26-09-2019",

year = "2019",

month = sep,

doi = "10.1109/CLUSTER.2019.8891000",

language = "English (US)",

series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019",

}

TY - GEN

T1 - Large-Scale Analysis of the Docker Hub Dataset

AU - Zhao, Nannan

AU - Tarasov, Vasily

AU - Albahar, Hadeel

AU - Anwar, Ali

AU - Rupprecht, Lukas

AU - Skourtis, Dimitrios

AU - Warke, Amit S.

AU - Mohamed, Mohamed

AU - Butt, Ali R.

PY - 2019/9

Y1 - 2019/9

N2 - Docker containers have become a prominent solution for supporting modern enterprise applications due to the highly desirable features of isolation, low overhead, and efficient packaging of the execution environment. Containers are created from images which are shared between users via a Docker registry. The amount of data Docker registries store is massive; for example, Docker Hub, a popular public registry, stores at least half a million public images. In this paper, we analyze over 167 TB of uncompressed Docker Hub images, characterize them using multiple metrics and evaluate the potential of file-level deduplication in Docker Hub. Our analysis helps to make conscious decisions when designing storage for containers in general and Docker registries in particular. For example, only 3% of the files in images are unique, which means file-level deduplication has a great potential to save storage space for the registry. Our findings can motivate and help improve the design of data reduction, caching, and pulling optimizations for registries.

AB - Docker containers have become a prominent solution for supporting modern enterprise applications due to the highly desirable features of isolation, low overhead, and efficient packaging of the execution environment. Containers are created from images which are shared between users via a Docker registry. The amount of data Docker registries store is massive; for example, Docker Hub, a popular public registry, stores at least half a million public images. In this paper, we analyze over 167 TB of uncompressed Docker Hub images, characterize them using multiple metrics and evaluate the potential of file-level deduplication in Docker Hub. Our analysis helps to make conscious decisions when designing storage for containers in general and Docker registries in particular. For example, only 3% of the files in images are unique, which means file-level deduplication has a great potential to save storage space for the registry. Our findings can motivate and help improve the design of data reduction, caching, and pulling optimizations for registries.

UR - http://www.scopus.com/inward/record.url?scp=85075269489&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85075269489&partnerID=8YFLogxK

U2 - 10.1109/CLUSTER.2019.8891000

DO - 10.1109/CLUSTER.2019.8891000

M3 - Conference contribution

AN - SCOPUS:85075269489

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

BT - Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019

Y2 - 23 September 2019 through 26 September 2019

ER -

Large-Scale Analysis of the Docker Hub Dataset

Abstract

Publication series

Conference

Bibliographical note

Publisher link

Find It

Other files and links

Fingerprint

Cite this