A workflow model for heterogeneous computing environments

This work presents the details of a model for fully decentralised scientific workflow management designed to support the efficient execution of applications in heterogeneous and unpredictable computing environments. The proposed model involves a novel application of the overlay metacomputer concept, built on a graph-oriented peer to peer framework. The use of peer to peer overlay on both batch and cycle scavenging systems simultaneously enables the aggregation and virtual homogenisation of heterogeneous resource collectives, and makes possible the execution of complex scientific workflow applications on otherwise batch oriented processors. Overlaying a peer to peer workflow management system on multiple independent sites decouples task expression from resource allocation, i.e., the host resource managers allocate processing elements to host the peers, allowing the peers execute multiple workflow tasks. This facilitates the efficient execution of very fine grained workflow expressions, which is shown to reduce both application makespan and wasted cycles when using processing elements allocated by a pre-emptive resource manager. Quantitative results are presented based on performance observed in a large scale heterogeneous multi-user environment.