New paper: Transparent Data Access for Scientific Workflows across Clouds
Discover a scientific workflow solution that pairs Onedata's global access with block-level transfers, moving only the data needed by remote jobs to reduce network traffic across clouds.
- Conference paper
- Authors
- Michał Orzechowski, Bartosz Baliś, Łukasz Dutka, Renata G. Słota, Jacek Kitowski
- Published in
- Euro-Par 2019: parallel processing workshops : Euro-Par 2019 international workshops : Göttingen, Germany, August 26–30, 2019 : revised selected papers, p. 751-755, ISSN 0302-9743
Abstract
We present a scientific workflow data management solution that combines global data access with a block-level optimization of data transfer, wherein only the data blocks that are used by a remote job are transferred over the network, significantly reducing data movement for specific common data access patterns. We propose the implementation of the solution based on the HyperFlow workflow management system and the Onedata data management platform. Preliminary results confirm the advantages of the proposed solution.