Transparent Data Access for Scientific Workflows across Clouds

Michał Orzechowski, Bartosz Baliś, Łukasz Dutka, Renata G. Słota, Jacek Kitowski

Euro-Par 2019: parallel processing workshops : Euro-Par 2019 international workshops : Göttingen, Germany, August 26–30, 2019 : revised selected papers, p. 751-755, ISSN 0302-9743

Show DOI Download PDF

Abstract

We present a scientific workflow data management solution that combines global data access with a block-level optimization of data transfer, wherein only the data blocks that are used by a remote job are transferred over the network, significantly reducing data movement for specific common data access patterns. We propose the implementation of the solution based on the HyperFlow workflow management system and the Onedata data management platform. Preliminary results confirm the advantages of the proposed solution.

Cite this article

@inproceedings{orzechowski_transparent_2020,
  title   = {Transparent Data Access for Scientific Workflows across Clouds},
  author  = {Michał Orzechowski and Bartosz Baliś and Łukasz Dutka and Renata G. Słota and Jacek Kitowski},
  booktitle = {Euro-Par 2019: parallel processing workshops : Euro-Par 2019 international workshops : Göttingen, Germany, August 26–30, 2019 : revised selected papers},
  editor  = {Ulrich Schwardmann and Christian Boehme and Dora B. Heras and Valeria Cardellini and Emmanuel Jeannot and Antonio Salis and Claudio Schifanella and Ravi Reddy Manumachu and Dieter Schwamborn and Laura Ricci and Oh Sangyoon and Thomas Gruber and Laura Antonelli and Stephen L. Scott},
  year    = {2020},
  pages   = {751--755},
  issn    = {0302-9743},
  doi     = {10.1007/978-3-030-48340-1_62},
}

Michał Orzechowski, Bartosz Baliś, Łukasz Dutka, Renata G. Słota, Jacek Kitowski,
Transparent Data Access for Scientific Workflows across Clouds,
Euro-Par 2019: parallel processing workshops : Euro-Par 2019 international workshops : Göttingen, Germany, August 26–30, 2019 : revised selected papers,
2020,
Pages 751-755,
ISSN 0302-9743,
https://doi.org/10.1007/978-3-030-48340-1_62