TY - GEN
T1 - Exploiting heterogeneity for collective data downloading in volunteer-based networks
AU - Kim, Jinoh
AU - Chandra, Abhishek
AU - Weissman, Jon
PY - 2007
Y1 - 2007
N2 - Scientific computing is being increasingly deployed over volunteer-based distributed computing environments consisting of idle resources on donated user machines. A fundamental challenge in these environments is the dissemination of data to the computation nodes, with the successful completion of jobs being driven by the efficiency of collective data download across compute nodes, and not only the individual download times. This paper considers the use of a data network consisting of data distributed across a set of data servers, and focuses on the server selection problem: how do individual nodes select a server for downloading data to minimize the communication makespan-the maximal download time for a data file. Through experiments conducted on a Pastry network running on PlanetLab, we demonstrate that nodes in a volunteer-based network are heterogeneous in terms of several metrics, such as bandwidth, load, and capacity, which impact their download behavior. We propose new server selection heuristics that incorporate these metrics, and demonstrate that these heuristics outperform traditional proximity-based server selection, reducing average makespans by at least 30%. We further show that incorporating information about download concurrency avoids overloading servers, and improves performance by about 17-43% over heuristics considering only proximity and bandwidth.
AB - Scientific computing is being increasingly deployed over volunteer-based distributed computing environments consisting of idle resources on donated user machines. A fundamental challenge in these environments is the dissemination of data to the computation nodes, with the successful completion of jobs being driven by the efficiency of collective data download across compute nodes, and not only the individual download times. This paper considers the use of a data network consisting of data distributed across a set of data servers, and focuses on the server selection problem: how do individual nodes select a server for downloading data to minimize the communication makespan-the maximal download time for a data file. Through experiments conducted on a Pastry network running on PlanetLab, we demonstrate that nodes in a volunteer-based network are heterogeneous in terms of several metrics, such as bandwidth, load, and capacity, which impact their download behavior. We propose new server selection heuristics that incorporate these metrics, and demonstrate that these heuristics outperform traditional proximity-based server selection, reducing average makespans by at least 30%. We further show that incorporating information about download concurrency avoids overloading servers, and improves performance by about 17-43% over heuristics considering only proximity and bandwidth.
UR - http://www.scopus.com/inward/record.url?scp=34548329800&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=34548329800&partnerID=8YFLogxK
U2 - 10.1109/CCGRID.2007.50
DO - 10.1109/CCGRID.2007.50
M3 - Conference contribution
AN - SCOPUS:34548329800
SN - 0769528333
SN - 9780769528335
T3 - Proceedings - Seventh IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2007
SP - 275
EP - 282
BT - Proceedings - Seventh IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2007
T2 - 7th IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2007
Y2 - 14 May 2007 through 17 May 2007
ER -