Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks

Feng Tian; Yang Zhang; Wei Ye; Cheng Jin; Ziyan Wu; Zhi Li Zhang

doi:10.1145/3482898.3483363

Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks

Feng Tian, Yang Zhang, Wei Ye, Cheng Jin, Ziyan Wu, Zhi Li Zhang

Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Scopus citations

Abstract

Data center networks (DCNs) have widely deployed RDMA to support data-intensive applications such as machine learning. While DCNs are designed with rich multi-path topology, current RDMA (hardware) technology does not support multi-path transport. In this paper we advance Maestro- a purely software-basedmulti-path RDMA solution - to effectively utilize the rich multi-path topology for load balancing and reliability. As a "middleware"operating at the user-space, Maestro is modulaR@and software-defined:Maestro decouples path selection and load balancing mechanisms from hardware features, and allows DCN operators and applications to make flexible decisions by employing the best mechanisms as needed. As such, Maestro can be readily deployed using existing RDMA hardware (NICs) to support distributed deep learning (DDL) applications. Our experiments show that Maestro is capable of fully utilizing multiple paths with negligible CPU overheads, thereby enhancing the performance of DDL applications.

Original language	English (US)
Title of host publication	SOSR 2021 - Proceedings of the 2021 ACM SIGCOMM Symposium on SDN Research
Publisher	Association for Computing Machinery, Inc
Pages	88-100
Number of pages	13
ISBN (Electronic)	9781450390842
DOIs	https://doi.org/10.1145/3482898.3483363
State	Published - Oct 11 2021
Event	2021 ACM SIGCOMM Symposium on SDN Research, SOSR 2021 - Virtual, Online, United States Duration: Sep 20 2021 → Sep 21 2021

Publication series

Name	Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR)

Conference

Conference	2021 ACM SIGCOMM Symposium on SDN Research, SOSR 2021
Country/Territory	United States
City	Virtual, Online
Period	9/20/21 → 9/21/21

Bibliographical note

Funding Information:
The research was supported in part by NSF under Grants CNS-1618339, CNS-1814322, CNS-1831140, CNS-1836772, CNS-1901103, CNS-2106771 and CCF-2123987.

Publisher Copyright:
© 2021 ACM.

Keywords

Data Center Networks
Distributed Deep Learning
Multi-Path RDMA

Publisher link

10.1145/3482898.3483363

Find It

Find It at U of M Libraries

Cite this

Tian, F., Zhang, Y., Ye, W., Jin, C., Wu, Z., & Zhang, Z. L. (2021). Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks. In SOSR 2021 - Proceedings of the 2021 ACM SIGCOMM Symposium on SDN Research (pp. 88-100). (Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR)). Association for Computing Machinery, Inc. https://doi.org/10.1145/3482898.3483363

Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks. / Tian, Feng; Zhang, Yang; Ye, Wei et al.
SOSR 2021 - Proceedings of the 2021 ACM SIGCOMM Symposium on SDN Research. Association for Computing Machinery, Inc, 2021. p. 88-100 (Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR)).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Tian, F, Zhang, Y, Ye, W, Jin, C, Wu, Z & Zhang, ZL 2021, Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks. in SOSR 2021 - Proceedings of the 2021 ACM SIGCOMM Symposium on SDN Research. Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR), Association for Computing Machinery, Inc, pp. 88-100, 2021 ACM SIGCOMM Symposium on SDN Research, SOSR 2021, Virtual, Online, United States, 9/20/21. https://doi.org/10.1145/3482898.3483363

@inproceedings{9dc75a881f24452ca8f11162078ea776,

title = "Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks",

abstract = "Data center networks (DCNs) have widely deployed RDMA to support data-intensive applications such as machine learning. While DCNs are designed with rich multi-path topology, current RDMA (hardware) technology does not support multi-path transport. In this paper we advance Maestro- a purely software-basedmulti-path RDMA solution - to effectively utilize the rich multi-path topology for load balancing and reliability. As a {"}middleware{"}operating at the user-space, Maestro is modulaR@and software-defined:Maestro decouples path selection and load balancing mechanisms from hardware features, and allows DCN operators and applications to make flexible decisions by employing the best mechanisms as needed. As such, Maestro can be readily deployed using existing RDMA hardware (NICs) to support distributed deep learning (DDL) applications. Our experiments show that Maestro is capable of fully utilizing multiple paths with negligible CPU overheads, thereby enhancing the performance of DDL applications.",

keywords = "Data Center Networks, Distributed Deep Learning, Multi-Path RDMA",

author = "Feng Tian and Yang Zhang and Wei Ye and Cheng Jin and Ziyan Wu and Zhang, {Zhi Li}",

note = "Funding Information: The research was supported in part by NSF under Grants CNS-1618339, CNS-1814322, CNS-1831140, CNS-1836772, CNS-1901103, CNS-2106771 and CCF-2123987. Publisher Copyright: {\textcopyright} 2021 ACM.; 2021 ACM SIGCOMM Symposium on SDN Research, SOSR 2021 ; Conference date: 20-09-2021 Through 21-09-2021",

year = "2021",

month = oct,

day = "11",

doi = "10.1145/3482898.3483363",

language = "English (US)",

series = "Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR)",

publisher = "Association for Computing Machinery, Inc",

pages = "88--100",

booktitle = "SOSR 2021 - Proceedings of the 2021 ACM SIGCOMM Symposium on SDN Research",

}

TY - GEN

T1 - Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks

AU - Tian, Feng

AU - Zhang, Yang

AU - Ye, Wei

AU - Jin, Cheng

AU - Wu, Ziyan

AU - Zhang, Zhi Li

N1 - Funding Information: The research was supported in part by NSF under Grants CNS-1618339, CNS-1814322, CNS-1831140, CNS-1836772, CNS-1901103, CNS-2106771 and CCF-2123987. Publisher Copyright: © 2021 ACM.

PY - 2021/10/11

Y1 - 2021/10/11

N2 - Data center networks (DCNs) have widely deployed RDMA to support data-intensive applications such as machine learning. While DCNs are designed with rich multi-path topology, current RDMA (hardware) technology does not support multi-path transport. In this paper we advance Maestro- a purely software-basedmulti-path RDMA solution - to effectively utilize the rich multi-path topology for load balancing and reliability. As a "middleware"operating at the user-space, Maestro is modulaR@and software-defined:Maestro decouples path selection and load balancing mechanisms from hardware features, and allows DCN operators and applications to make flexible decisions by employing the best mechanisms as needed. As such, Maestro can be readily deployed using existing RDMA hardware (NICs) to support distributed deep learning (DDL) applications. Our experiments show that Maestro is capable of fully utilizing multiple paths with negligible CPU overheads, thereby enhancing the performance of DDL applications.

AB - Data center networks (DCNs) have widely deployed RDMA to support data-intensive applications such as machine learning. While DCNs are designed with rich multi-path topology, current RDMA (hardware) technology does not support multi-path transport. In this paper we advance Maestro- a purely software-basedmulti-path RDMA solution - to effectively utilize the rich multi-path topology for load balancing and reliability. As a "middleware"operating at the user-space, Maestro is modulaR@and software-defined:Maestro decouples path selection and load balancing mechanisms from hardware features, and allows DCN operators and applications to make flexible decisions by employing the best mechanisms as needed. As such, Maestro can be readily deployed using existing RDMA hardware (NICs) to support distributed deep learning (DDL) applications. Our experiments show that Maestro is capable of fully utilizing multiple paths with negligible CPU overheads, thereby enhancing the performance of DDL applications.

KW - Data Center Networks

KW - Distributed Deep Learning

KW - Multi-Path RDMA

UR - http://www.scopus.com/inward/record.url?scp=85119432621&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85119432621&partnerID=8YFLogxK

U2 - 10.1145/3482898.3483363

DO - 10.1145/3482898.3483363

M3 - Conference contribution

AN - SCOPUS:85119432621

T3 - Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR)

SP - 88

EP - 100

BT - SOSR 2021 - Proceedings of the 2021 ACM SIGCOMM Symposium on SDN Research

PB - Association for Computing Machinery, Inc

T2 - 2021 ACM SIGCOMM Symposium on SDN Research, SOSR 2021

Y2 - 20 September 2021 through 21 September 2021

ER -

Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

CNS Core:Meidium: NFLambda -- A Granular, Scalable and Secure NFV Framework for High Performance Packet Processing at 100 Gbps and Beyond

FMitF: Track I: ComScaN: Composition and Scaling of Network Service Functions

CNS Core: Medium: Collaborative: Exploring and Exploiting Learning for Efficient Network Control: Non-Stationarity, Inter-Dependence, and Domain-Knowledge

Cite this

Accelerating Distributed Deep Learning using Multi-Path RDMA in Data Center Networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Projects

CNS Core:Meidium: NFLambda -- A Granular, Scalable and Secure NFV Framework for High Performance Packet Processing at 100 Gbps and Beyond

FMitF: Track I: ComScaN: Composition and Scaling of Network Service Functions

CNS Core: Medium: Collaborative: Exploring and Exploiting Learning for Efficient Network Control: Non-Stationarity, Inter-Dependence, and Domain-Knowledge

Cite this