Harmonizing Sequential and Random Access to Datasets in Organizationally Distributed Environments

Michał Wrzeszcz, Łukasz Opioła, Bartosz Kryza, Łukasz Dutka, Renata G. Słota, Jacek Kitowski

Computational Science – ICCS 2019 : 19th International Conference : Faro, Portugal, June 12–14, 2019 : proceedings, LNCS Pt. 1, p. 295-308, ISSN 0302-9743, ISBN 978-3-030-22734-0

Show DOI Download PDF

Abstract

Computational science is rapidly developing, which pushes the boundaries in data management concerning the size and structure of datasets, data processing patterns, geographical distribution of data and performance expectations. In this paper we present a solution for harmonizing data access performance, i.e. finding a compromise between local and remote read/write efficiency that would fit those evolving requirements. It is based on variable-size logical data-chunks (in contrast to fixed-size blocks), direct storage access and several mechanisms improving remote data access performance. The solution is implemented in the Onedata system and suited to its multi-layer architecture, supporting organizationally distributed environments – with limited trust between data providers. The solution is benchmarked and compared to XRootD + XCache, which offers similar functionalities. The results show that the performance of both systems is comparable, although overheads in local data access are visibly lower in Onedata.

Cite this article

@inproceedings{wrzeszcz_harmonizing_2019,
  title   = {Harmonizing Sequential and Random Access to Datasets in Organizationally Distributed Environments},
  author  = {Michał Wrzeszcz and Łukasz Opioła and Bartosz Kryza and Łukasz Dutka and Renata G. Słota and Jacek Kitowski},
  booktitle = {Computational Science – ICCS 2019 : 19th International Conference : Faro, Portugal, June 12–14, 2019 : proceedings, LNCS Pt. 1},
  editor  = {João M. F. Rodrigues and Pedro J. S. Cardoso and Jânio Monteiro and Roberto Lam and Valeria V. Krzhizhanovskaya and Michael H. Lees and Jack J. Dongarra and Peter M.A. Sloot},
  year    = {2019},
  pages   = {295--308},
  issn    = {0302-9743},
  isbn    = {978-3-030-22734-0},
  doi     = {10.1007/978-3-030-22734-0_22},
}

Michał Wrzeszcz, Łukasz Opioła, Bartosz Kryza, Łukasz Dutka, Renata G. Słota, Jacek Kitowski,
Harmonizing Sequential and Random Access to Datasets in Organizationally Distributed Environments,
Computational Science – ICCS 2019 : 19th International Conference : Faro, Portugal, June 12–14, 2019 : proceedings, LNCS Pt. 1,
2019,
Pages 295-308,
ISSN 0302-9743,
ISBN 978-3-030-22734-0,
https://doi.org/10.1007/978-3-030-22734-0_22