New paper: Harmonizing Sequential and Random Access to Datasets in Organizationally Distributed Environments

See how Onedata harmonizes sequential and random dataset access with variable-size chunks, direct storage access, and remote-read optimizations while achieving competitive performance against XRootD and XCache.

Conference paper
Authors: Michał Wrzeszcz, Łukasz Opioła, Bartosz Kryza, Łukasz Dutka, Renata G. Słota, Jacek Kitowski
Published in: Computational Science – ICCS 2019 : 19th International Conference : Faro, Portugal, June 12–14, 2019 : proceedings, LNCS Pt. 1, p. 295-308, ISSN 0302-9743, ISBN 978-3-030-22734-0
DOI: 10.1007/978-3-030-22734-0_22; Full details & cite →

Abstract

Computational science is rapidly developing, which pushes the boundaries in data management concerning the size and structure of datasets, data processing patterns, geographical distribution of data and performance expectations. In this paper we present a solution for harmonizing data access performance, i.e. finding a compromise between local and remote read/write efficiency that would fit those evolving requirements. It is based on variable-size logical data-chunks (in contrast to fixed-size blocks), direct storage access and several mechanisms improving remote data access performance. The solution is implemented in the Onedata system and suited to its multi-layer architecture, supporting organizationally distributed environments – with limited trust between data providers. The solution is benchmarked and compared to XRootD + XCache, which offers similar functionalities. The results show that the performance of both systems is comparable, although overheads in local data access are visibly lower in Onedata.