Global Access to Legacy Data-Sets in Multi-Cloud Applications with Onedata

Michał Orzechowski, Michał Wrzeszcz, Bartosz Kryza, Łukasz Dutka, Renata G. Słota, Jacek Kitowski

Parallel Processing and Applied Mathematics : 14th international Conference, PPAM 2022 : Gdansk, Poland, September 11–14, 2022 : revised selected papers, Pt. 1, vol. 13826, p. 305-317, ISSN 0302-9743, ISBN 978-3-031-30442-2

Show DOI

Abstract

Data access and management for multi-cloud applications proves challenging where there is a need for efficient access to large pre-existing, legacy data-sets. To address this problem, we created an indexing subsystem incorporated into Onedata data management system achieving a global multi-cloud integration of legacy storage systems containing large data-sets. The solution is based on metadata management, organization and periodic monitoring of legacy data-sets, what makes possible scheme-less co-existence a legacy storage system and Onedata for managing the same data collections. Thanks to block based data transfer provided by Onedata large files stored on legacy storage systems can be efficiently accessed from the cloud. The approach has been initially evaluated in a multi-cloud deployment scenario, where a large data collection stored on legacy storage system located in a super-computing center is processed on a commercial cloud.

Cite this article

@inproceedings{orzechowski_global_2023,
  title   = {Global Access to Legacy Data-Sets in Multi-Cloud Applications with Onedata},
  author  = {Michał Orzechowski and Michał Wrzeszcz and Bartosz Kryza and Łukasz Dutka and Renata G. Słota and Jacek Kitowski},
  booktitle = {Parallel Processing and Applied Mathematics : 14th international Conference, PPAM 2022 : Gdansk, Poland, September 11–14, 2022 : revised selected papers, Pt. 1},
  editor  = {Roman Wyrzykowski and Jack Dongarra and Ewa Deelman and Konrad Karczewski},
  year    = {2023},
  volume  = {13826},
  pages   = {305--317},
  issn    = {0302-9743},
  isbn    = {978-3-031-30442-2},
  doi     = {10.1007/978-3-031-30442-2_23},
}

Michał Orzechowski, Michał Wrzeszcz, Bartosz Kryza, Łukasz Dutka, Renata G. Słota, Jacek Kitowski,
Global Access to Legacy Data-Sets in Multi-Cloud Applications with Onedata,
Parallel Processing and Applied Mathematics : 14th international Conference, PPAM 2022 : Gdansk, Poland, September 11–14, 2022 : revised selected papers, Pt. 1,
2023,
Volume 13826,
Pages 305-317,
ISSN 0302-9743,
ISBN 978-3-031-30442-2,
https://doi.org/10.1007/978-3-031-30442-2_23