WASP: Wide-area Adaptive Stream Processing

Albert Jonathan, Abhishek Chandra, Jon Weissman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Adaptability is critical for stream processing systems to ensure stable, low-latency, and high-throughput processing of long-running queries. Such adaptability is particularly challenging for wide-area stream processing due to the highly dynamic nature of the wide-area environment, which includes unpredictable workload patterns, variable network bandwidth, occurrence of stragglers, and failures. Unfortunately, existing adaptation techniques typically achieve these performance goals by compromising the quality/accuracy of the results, and they are often application-dependent. In this work, we rethink the adaptability property of wide-area stream processing systems and propose a resource-aware adaptation framework, called WASP. WASP adapts queries through a combination of multiple techniques: task re-assignment, operator scaling, and query re-planning, and applies them in a WAN-aware manner. It is able to automatically determine which adaptation action to take depending on the type of queries, dynamics, and optimization goals. We have implemented a WASP prototype on Apache Flink. Experimental evaluation with the YSB benchmark and a real Twitter trace shows that WASP can handle various dynamics without compromising the quality of the results.

Original languageEnglish (US)
Title of host publicationMiddleware 2020 - Proceedings of the 2020 21st International Middleware Conference
PublisherAssociation for Computing Machinery, Inc
Pages221-235
Number of pages15
ISBN (Electronic)9781450381536
DOIs
StatePublished - Dec 7 2020
Event21st International Middleware Conference, Middleware 2020 - Virtual, Online, Netherlands
Duration: Dec 7 2020Dec 11 2020

Publication series

NameMiddleware 2020 - Proceedings of the 2020 21st International Middleware Conference

Conference

Conference21st International Middleware Conference, Middleware 2020
Country/TerritoryNetherlands
CityVirtual, Online
Period12/7/2012/11/20

Bibliographical note

Funding Information:
The authors would like to thank the anonymous Middleware reviewers for their valuable comments and feedback. The work is supported by grant NSF CNS-1619254 and CNS-1717834.

Publisher Copyright:
© 2020 Association for Computing Machinery.

Fingerprint

Dive into the research topics of 'WASP: Wide-area Adaptive Stream Processing'. Together they form a unique fingerprint.

Cite this