The Circulate architecture: Avoiding workflow bottlenecks caused by centralised orchestration

Adam Barker, Jon B. Weissman, Jano I. van Hemert

Research output: Contribution to journalArticlepeer-review

16 Scopus citations

Abstract

As the number of services and the size of data involved in workflows increases, centralised orchestration techniques are reaching the limits of scalability. In the classic orchestration model, all data passes through a centralised engine, which results in unnecessary data transfer, wasted bandwidth and the engine to become a bottleneck to the execution of a workflow. This paper presents and evaluates the Circulate architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Circulate could be realised within any existing workflow framework, in this paper, we focus on WS-Circulate, a Web services based implementation. Taking inspiration from the Montage workflow, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 24 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using the Circulate architecture increases as the complexity of a workflow grows.

Original languageEnglish (US)
Pages (from-to)221-235
Number of pages15
JournalCluster Computing
Volume12
Issue number2 SPEC. ISS.
DOIs
StatePublished - 2009

Bibliographical note

Funding Information:
in Mathematics and Physical Sci- ences from the Leiden University (2002), The Netherlands. He is a Research Associate in the School of Informatics of the University of Edinburgh and a visiting researcher at the Human Genetics Unit in Ed- inburgh of the United Kingdom’s Medical Research Council. He is re- sponsible for leading the research within the National e-Science Cen- tre. He has held research positions at the Leiden University (NL), the Vienna University of Technology (AUT) and the National Research Institute for Mathematics and Computer Science (NL). In 2004, he was awarded the talented young researcher fellowship by the Netherlands Organization for Scientific Research. Many of his research projects have included partners from industry.

Keywords

  • Decentralised orchestration
  • Web services
  • Workflow
  • Workflow optimisation

Fingerprint

Dive into the research topics of 'The Circulate architecture: Avoiding workflow bottlenecks caused by centralised orchestration'. Together they form a unique fingerprint.

Cite this