Accelerated Training via Device Similarity in Federated Learning

Yuanli Wang, Joel Wolfrath, Nikhil Sreekumar, Dhruv Kumar, Abhishek Chandra

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Federated Learning is a privacy-preserving, machine learning technique that generates a globally shared model with in-situ model training on distributed devices. These systems are often comprised of millions of user devices and only a subset of available devices can be used for training in each epoch. Designing a device selection strategy is challenging, given that devices are highly heterogeneous in both their system resources and training data. This heterogeneity makes device selection very crucial for timely model convergence and sufficient model accuracy. Existing approaches have addressed system heterogeneity for device selection but have largely ignored the data heterogeneity. In this work, we analyze the impact of data heterogeneity on device selection, model convergence, model accuracy, and fault tolerance in a federated learning setting. Based on our analysis, we propose that clustering devices with similar data distributions followed by selecting the devices with the best processing capacity from each cluster can significantly improve the model convergence without compromising model accuracy. This clustering also guides us in designing policies for fault tolerance in the system. We propose three methods for identifying groups of devices with similar data distributions. We also identify and discuss rich trade-offs between privacy, bandwidth consumption, and computation overhead for each of these proposed methods. Our preliminary experiments show that the proposed methods can provide a 46%-58% reduction in training time compared to existing approaches in reaching the same accuracy.

Original languageEnglish (US)
Title of host publicationEdgeSys 2021 - Proceedings of the 4th International Workshop on Edge Systems, Analytics and Networking, Part of EuroSys 2021
PublisherAssociation for Computing Machinery, Inc
Pages31-36
Number of pages6
ISBN (Electronic)9781450382915
DOIs
StatePublished - Apr 26 2021
Event4th International Workshop on Edge Systems, Analytics and Networking, EdgeSys 2021, in conjunction with ACM EuroSys 2021 - Virtual, Online, United Kingdom
Duration: Apr 26 2021 → …

Publication series

NameEdgeSys 2021 - Proceedings of the 4th International Workshop on Edge Systems, Analytics and Networking, Part of EuroSys 2021

Conference

Conference4th International Workshop on Edge Systems, Analytics and Networking, EdgeSys 2021, in conjunction with ACM EuroSys 2021
CountryUnited Kingdom
CityVirtual, Online
Period4/26/21 → …

Bibliographical note

Funding Information:
This research was supported in part by the NSF under grant CNS-1717834.

Publisher Copyright:
© 2021 ACM.

Fingerprint Dive into the research topics of 'Accelerated Training via Device Similarity in Federated Learning'. Together they form a unique fingerprint.

Cite this