Performance-based path determination for interprocessor communication in distributed computing systems

Jun Seonq Kim, David J. Lilja

Research output: Contribution to journalArticlepeer-review

8 Scopus citations


The different types of messages used by a parallel application program executing in a distributed computing system can each have unique characteristics so that no single, communication network can produce the lowest latency for all messages. For instance, short control messages may be sent with the lowest overhead on one type of network, such as Ethernet, while bulk data transfers may be better suited to a different type of network, such as Fibre Channel or HiPPI. This work investigates how to exploit multiple heterogeneous communication networks that interconnect the same set of processing nodes using a set of techniques we call performance-based path determination (PBPD). The performance-based path selection (PBPS) technique selects the best (lowest latency) network among several for each individual message to reduce the communication overhead of parallel programs. The performance-based path aggregation (PBPA) technique, on the other hand, aggregates multiple networks into a single virtual network to increase the available bandwidth. We test the PBPD techniques on a cluster of SGI multiprocessors interconnected with Ethernet, Fibre Channel, and HiPPI networks using a custom communication library built on top of the TCP/IP protocol layers. We find that PBPS can reduce communication overhead in applications compared to using either network alone, while aggregating networks into a single virtual network can reduce communication latency for bandwidth-limited applications. The performance of the PBPD techniques depends on the mix of message sizes in the application program and the relative overheads of the networks, as demonstrated in our analytical models.

Original languageEnglish (US)
Pages (from-to)316-327
Number of pages12
JournalIEEE Transactions on Parallel and Distributed Systems
Issue number3
StatePublished - 1999

Bibliographical note

Funding Information:
This work was supported in part by the U.S. National Science Foundation under Grant CDA-9414015 and by a University of Minnesota McKnight Land-Grant Professorship. Preliminary versions of this work were presented at the 1997 Heterogeneous Computing Workshop [9] and the 1997 International Symposium on High Performance Distributed Computing [10].


Dive into the research topics of 'Performance-based path determination for interprocessor communication in distributed computing systems'. Together they form a unique fingerprint.

Cite this