MESH: A flexible distributed hypergraph processing system

Benjamin Heintz, Rankyung Hong, Shivangi Singh, Gaurav Khandelwal, Corey Tesdahl, Abhishek Chandra

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

With the rapid growth of large online social networks, the ability to analyze large-scale social structure and behavior has become critically important, and this has led to the development of several scalable graph processing systems. In reality, however, social interaction takes place not only between pairs of individuals as in the graph model, but rather in the context of multi-user groups. Research has shown that such group dynamics can be better modeled through a more general hypergraph model, resulting in the need to build scalable hypergraph processing systems. In this paper, we present MESH, a flexible distributed framework for scalable hypergraph processing. MESH provides an easy-to-use and expressive application programming interface that naturally extends the 'think like a vertex' model common to many popular graph processing systems. Our framework provides a flexible implementation based on an underlying graph processing system, and enables different design choices for the key implementation issues of partitioning a hypergraph representation. We implement MESH on top of the popular GraphX graph processing framework in Apache Spark. Using a variety of real datasets and experiments conducted on a local 8-node cluster as well as a 65-node Amazon AWS testbed, we demonstrate that MESH provides flexibility based on data and application characteristics, as well as scalability with cluster size. We further show that it is competitive in performance to HyperX, another hypergraph processing system based on Spark, while providing a much simpler implementation (requiring about 5X fewer lines of code), thus showing that simplicity and flexibility need not come at the cost of performance.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE International Conference on Cloud Engineering, IC2E 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages12-22
Number of pages11
ISBN (Electronic)9781728102184
DOIs
StatePublished - Jun 2019
Event7th IEEE International Conference on Cloud Engineering, IC2E 2019 - Prague, Czech Republic
Duration: Jun 24 2019Jun 27 2019

Publication series

NameProceedings - 2019 IEEE International Conference on Cloud Engineering, IC2E 2019

Conference

Conference7th IEEE International Conference on Cloud Engineering, IC2E 2019
CountryCzech Republic
CityPrague
Period6/24/196/27/19

    Fingerprint

Keywords

  • Distributed Processing System
  • Hypergraph Processing System

Cite this

Heintz, B., Hong, R., Singh, S., Khandelwal, G., Tesdahl, C., & Chandra, A. (2019). MESH: A flexible distributed hypergraph processing system. In Proceedings - 2019 IEEE International Conference on Cloud Engineering, IC2E 2019 (pp. 12-22). [8790188] (Proceedings - 2019 IEEE International Conference on Cloud Engineering, IC2E 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IC2E.2019.00-11