Adaptability is critical for stream processing systems to ensure stable, low-latency, and high-throughput processing of long-running queries. Such adaptability is particularly challenging for wide-area stream processing due to the highly dynamic nature of the wide-area environment, which includes unpredictable workload patterns, variable network bandwidth, occurrence of stragglers, and failures. Unfortunately, existing adaptation techniques typically achieve these performance goals by compromising the quality/accuracy of the results, and they are often application-dependent. In this work, we rethink the adaptability property of wide-area stream processing systems and propose a resource-aware adaptation framework, called WASP. WASP adapts queries through a combination of multiple techniques: task re-assignment, operator scaling, and query re-planning, and applies them in a WAN-aware manner. It is able to automatically determine which adaptation action to take depending on the type of queries, dynamics, and optimization goals. We have implemented a WASP prototype on Apache Flink. Experimental evaluation with the YSB benchmark and a real Twitter trace shows that WASP can handle various dynamics without compromising the quality of the results.
|Original language||English (US)|
|Title of host publication||Middleware 2020 - Proceedings of the 2020 21st International Middleware Conference|
|Publisher||Association for Computing Machinery, Inc|
|Number of pages||15|
|State||Published - Dec 7 2020|
|Event||21st International Middleware Conference, Middleware 2020 - Virtual, Online, Netherlands|
Duration: Dec 7 2020 → Dec 11 2020
|Name||Middleware 2020 - Proceedings of the 2020 21st International Middleware Conference|
|Conference||21st International Middleware Conference, Middleware 2020|
|Period||12/7/20 → 12/11/20|
Bibliographical noteFunding Information:
The authors would like to thank the anonymous Middleware reviewers for their valuable comments and feedback. The work is supported by grant NSF CNS-1619254 and CNS-1717834.
© 2020 Association for Computing Machinery.