File popularity

A built-in database view that allows tracking of usage statistics for files in a space. Lays at the foundation of the auto-cleaning mechanism. For more details, consult the documentation.

centered

Auto-cleaning

The purpose of the auto-cleaning mechanism is to automatically maintain storage usage at a certain level and ensure that there is enough space for new replicas when performing continuous computations. The process is safe — only redundant replicas (duplicated on remote providers) are evicted. For more details, consult the documentation.

centered

LUMA

LUMA DB (Local User MApping) is a database that stores mappings between Onedata user accounts and local user accounts/credentials on storage backends, e.g.:

  • POSIX UID/GID
  • Ceph username/password
  • S3 accessKey/secretKey

The LUMA DB can be set up for each storage registered in Oneprovider separately, with different credentials for users on each storage.

Credential types

centered

Acquiring mappings on non-imported storage

centered

Acquiring mappings on imported storage

centered

Mapping imported NFSv4 ACLs

centered

Configuration

LUMA DB is initially empty for new supports and must be filled with mappings using one of the 3 LUMA feeds:

  1. auto — mappings are added as needed using an automatic algorithm:
    • POSIX-like: randomized UIDs
    • other: root credentials
  2. local — all mappings in LUMA DB must be set explicitly by the Oneprovider admin using the REST API. The mappings must be set before inviting users to join the space, otherwise, their operations on files will fail.
  3. external — LUMA DB is populated by lazily querying an external microservice that implements a standardized REST API. The Oneprovider admin is responsible for the implementation and maintenance of the server. This approach offers great elasticity and allows integration with systems like LDAP.

LUMA DB configuration is part of the storage management GUI: centered

NOTE: Modification of the type of feed for LUMA DB results in automatic deletion of all entries in the LUMA DB for given storage resource. Newly set feed will be used to populate LUMA DB once again.

WARNING: Clearing the LUMA DB when feed type is changed means that the local feed is also cleared. In order to use the local feed then, all mappings must be added once again.

Practice: regular (non-imported) POSIX storage

  1. Create a new storage backend called "posix-local-luma"
    • POSIX type
    • choose an arbitrary (but existing and empty) mount point

      remember that the hosts's filesystem is available under /hostfs

    • make sure to check local LUMA feed
  2. Create a space "luma-test" and support it with the newly created storage.

  1. Insert a user mapping into the LUMA DB:

    NOTE: make sure to export the following variables or replace them in the snippet: TOKEN, PANEL_HOST, STORAGE_ID, USER_ID (check the Manage account tab in GUI)

    curl -H "X-Auth-Token: $TOKEN" -X POST \
https://$PANEL_HOST:9443/api/v3/onepanel/provider/storages/$STORAGE_ID/luma/local_feed/storage_access/all/onedata_user_to_credentials \
-H 'Content-type: application/json' \
-d '{
    "onedataUser": {
        "mappingScheme": "onedataUser",
        "onedataUserId": "'$USER_ID'"
    },
    "storageUser": {
        "storageCredentials": {
            "type": "posix",
            "uid": 1300
        },
        "displayUid": 1500
    }
}'

    NOTE: On POSIX compatible storage backends (currently POSIX and NFS), credentials consist of 2 integers: UID and GID. Only the UID field can be assigned in LUMA. The GID is constant for the space (all space members are treated as the owner group). This strategy ensures that all files created in the space have the same GID owner on the supporting storage so that ownership of the files in the space is correctly mapped on the storage.

  1. Upload a file into the space.

  2. Verify file credentials on the storage (list the mount point):

    ~$ ls -l
total 112
-rw-rw-r-- 1 1300 1000 113068 Nov 24 13:05 example.txt

  3. Verify file display credentials inside a Oneclient mount:

    ~$ ls -l
total 112
-rw-rw-r-- 1 1500 1000 113068 Nov 24 13:05 example.txt

 

For more information about LUMA DB, consult the documentation.

Next chapter:

Space Marketplace — practice