# Local User Mapping — LUMA

# Overview

LUMA is a database that stores mappings between Onedata user accounts and local user accounts/credentials on storage resources, e.g.:

  • POSIX UID/GID
  • Ceph username and password
  • GlusterFS UID/GID

LUMA DB essentially establishes a relation between members of a Onedata space and user accounts recognized by different storage providers, reflecting the ownership of user data kept on various storage systems in different locations. The LUMA DB can be set up for each storage registered in Oneprovider separately, with different credentials for users on each storage.

To understand LUMA DB and how mappings should be defined, consult the POSIX-like file ownership and permissions section.

# Storage and LUMA essentials

The below checklist contains the fundamental steps to properly configure your storage and the LUMA DB. Nonetheless, it is recommended that you first familiarize yourself with all the concepts described in this chapter.

# Checklist for regular (non-imported) storage backends

  1. Choose the right LUMA DB feed.
  2. Substantiate the required mappings (if applicable).
  3. Make sure that storage credentials to which Onedata users will be mapped exist and are reflected in the storage system.

# Checklist for imported storage backends

  1. Choose the right LUMA DB feed.
  2. Substantiate the required mappings (if applicable).
  3. Make sure that storage credentials to which Onedata users will be mapped exist and are reflected in the storage system.
  4. As far as possible, ensure that all files on the imported storage have the same group owner (GID) — read more.
  5. If you intend to enable auto storage import with the enabled Synchronize ACL option you must properly configure the chosen LUMA feed to deliver mappings for NFSv4 ACL users and groups for which ACLs are set on the imported storage.

# Configuration

Configuration of LUMA DB is part of a storage configuration.

It is possible to choose the type of feed for populating the DB. In the case of selecting the external feed, it is compulsory to set the URL of the external server that will be lazily queried to populate the LUMA DB. It is also possible to set an API key that will be sent with each request to the external server. An example configuration of the LUMA feed is presented in the below picture:

screen-luma-config

NOTE: Modification of the type of feed for LUMA DB results in automatic deletion of all entries in the LUMA DB for given storage resource. Newly set feed will be used to populate LUMA DB once again.

WARNING: Clearing the LUMA DB when feed type is changed means that the local feed is also cleared. In order to use the local feed then, all mappings must be added once again.

# Credential types

These definitions are used throughout the rest of the chapter.

# Storage credentials

Credentials for the corresponding storage backend, which are used to perform operations on the storage in the context of a specific user.

# Display credentials

Credentials used to present POSIX credentials (UID & GID) of a file owner in the file's attributes, e.g., they are displayed in the result of ls operation in Oneclient or when fetching file attributes using REST API.

# Onedata user credentials

Credentials identifying a user in the Onedata system.

# Onedata group credentials

Credentials identifying a group in the Onedata system.

# Database

LUMA DB consists of 5 tables that store 4 types of records, as described below.

# Tables

Table Key Record
Storage users (StorageId, UserId) StorageUser
Spaces POSIX storage defaults (StorageId, SpaceId) PosixCredentials
Spaces display defaults (StorageId, SpaceId) PosixCredentials
Onedata users (StorageId, UID/AclUser) OnedataUser
Onedata groups (StorageId, AclGroup) OnedataGroup

# Records

# StorageUser

This record is used to represent the user on the specific storage backend and consists of two fields:

  • storageCredentials is a map storing credentials relevant for the storage backend,
  • displayUid is a field used to present the corresponding user as the owner of a file (UID) in Oneclient or GUI
{
    "storageCredentials": JSON, // Map storing credentials relevant for given storage backend
    "displayUid": Integer
}

# PosixCredentials

This record stores standard POSIX-like user ID and group ID.

{
    "uid": Integer,
    "gid": Integer
}

# OnedataUser

This record represents credentials identifying a user in the Onedata system. The record has one of the two schemes:

  • "onedataUser" — the user's ID is stored directly in the record,
  • "idpUser" — the record stores the ID of an external identity provider (IdP) and the ID of the user understood by the IdP.
{
    "mappingScheme": "onedataUser" | "idpUser",
    // fields for "onedataUser" scheme
    "onedataUserId": String,
    // fields for "idpUser" scheme
    "idp": String,
    "subjectId": String
}

For more information on mapping users from an external IdP to Onedata users, see here.

# OnedataGroup

This record represents credentials identifying a group in the Onedata system. The record has one of the two schemes:

  • "onedataGroup" — group's ID is stored directly in the record,
  • "idpEntitlement" — the record stores the ID of an external identity provider (IdP) and the ID of the group understood by the IdP.
{
    "mappingScheme": "onedataGroup" | "idpEntitlement",
    // fields for "onedataGroup" scheme
    "onedataGroupId": String,
    // fields for "idpEntitlement" scheme
    "idp": String,
    "idpEntitlement": String
}

For more information on mapping groups from external IdP to Onedata groups, see here.

# LUMA mappings

The mappings that need to be provided are different for regular and imported storage backends, as shown below. Instructions on how to populate the LUMA DB with required mappings can be found in the next section.

# Regular (non-imported) storage backends

If the storage import is disabled, LUMA is used to access the storage in the user's context, and it is sufficient to provide the storage credentials mapping and recommended setting the default space GID in case of POSIX storage (see below). The display credentials are optional — LUMA will use defaults if not provided.

# Onedata user to storage credentials mapping

Storage credentials are acquired differently depending on the storage type.

On POSIX incompatible storage backends, field storageCredentials from StorageUser record stored in Storage users table is returned.

On POSIX-compatible storage backends (currently POSIX, GlusterFS, NullDevice), credentials consist of 2 integers: UID and GID. Only the UID field is stored in the StorageUser record, the GID is constant for the space (all space members are treated as the owner group — read more). This strategy ensures that all files created in the space have the same GID owner on the supporting storage so that ownership of the files in the space is correctly mapped on the storage.

Due to the above reasons, GID is acquired from field gid from PosixCredentials record stored in Spaces POSIX storage defaults table.

NOTE: To enable use of Oneclient in Direct I/O mode for all space users, the storage admin has to ensure that:

  • LUMA properly maps each Onedata user to an UID that is recognized in the system hosting the storage
  • There exists a group in the system hosting the storage with GID equal to the virtual space GID (read more), and all above-mentioned UIDs belong to the group

# Onedata user to display credentials mapping

Display credentials are POSIX-compatible credentials that consist of 2 integers: UID and GID.

UID is acquired from field displayUid from StorageUser record stored in Storage users table.

GID is acquired from field gid from PosixCredentials record stored in Spaces display defaults table.

# Imported storage backends

The below mappings are associated with the concept of storage import and should only be considered when the corresponding storage is an imported storage.

  • mapping storage user to Onedata user — used in case of importing files from storage. It allows mapping the owner of the storage file to a specific Onedata user who will become the owner of the file imported to the space. Storage user is identified by the value of the UID field returned from stat operation or equivalent on the storage backend.
  • mapping storage NFSv4 ACL principal to Onedata user/group — used in case of importing files from storage that supports NFSv4 ACLs (opens new window), with Synchronize ACL option enabled. It allows mapping ACL principal to a specific user/group in the Onedata. If the Synchronize ACL option is disabled, this mapping does not have to be defined.

WARNING: It is possible that imported files have different GIDs. Oneprovider does not attempt to map them to the Onedata groups model as it's not compatible with the POSIX groups model. It is strongly recommended that admin of the legacy storage to be imported ensures that the file structure is compliant with the Onedata model — all files in the space should have the same group owner. Otherwise, access to imported files may be denied by the underlying storage system.

# Storage user to Onedata user mapping

UID returned from stat operation or equivalent on given storage backend is mapped to OnedataUser record stored in the Onedata users table. Information stored in the record allows identifying the corresponding Onedata user. If the mapping is not defined, the virtual space user becomes the owner of the imported file.

NOTE: This mapping is used only in case of enabled auto storage import.

# Storage ACL user to Onedata user mapping

ACL principal is mapped to OnedataUser record stored in the Onedata users table. Information stored in the record allows identifying the corresponding Onedata user. If the mapping is not defined, importing the file will return an error.

NOTE: This mapping is used only in case of enabled auto storage import with Synchronize ACL option enabled.

# Storage ACL group to Onedata group mapping

ACL group principal is mapped to OnedataGroup record stored in the Onedata groups table. Information stored in the record allows identifying the corresponding Onedata group. If the mapping is not defined, importing the file will return an error.

NOTE: This mapping is used only in case of enabled auto storage import with Synchronize ACL option enabled.

# Populating LUMA DB with feeds

LUMA DB can be filled with mappings in 3 different ways, called feeds:

# Auto Feed

Auto feed populates LUMA DB using an automatic algorithm. This section describes the algorithm for each table.

NOTE: If you want to fully configure LUMA DB for efficient and secure storage access you should choose local or external feeds. Auto feed is meant to be used at the very beginning of learning how to maintain Oneprovider service to ease the process of the first deployment.

# Auto feed for Storage users table

This table is populated differently depending on the storage type for which it is configured. In the case of POSIX-compatible storage backends, UID is generated based on the Onedata user ID.

NOTE: Using the auto feed on POSIX-compatible storage backends can make it impossible to use Oneclient in Direct I/O mode, as the automatically generated mappings will not correspond to actual UIDs/GIDs recognized by the storage system.

The same UID is used as the user's displayUid.

In case of POSIX incompatible storage backends, the storage credentials of the storage admin are used as the user's storage credentials.

WARNING: Allowing all space members to access storage with admin credentials is a potential security risk. Therefore, using the auto feed is not recommended for POSIX incompatible storage backends.

displayUid is generated the same way as UID in the case of POSIX-compatible storage.

# Auto feed for Spaces POSIX storage defaults table

This table is populated only for POSIX-compatible storage backends. UID and GID of the owner of the storage mount-point are used, as returned by the stat operation.

# Auto feed for Spaces display defaults table

This table is populated differently depending on the storage type for which it is configured. In the case of POSIX-compatible storage backends, the same credentials as stored in Spaces POSIX storage defaults table are used.

In the case of POSIX incompatible storage backends, UID and GID are generated based on the ID of the space for which the table is populated.

# Auto feed for Onedata users table

This table can be populated automatically only with entries that associate the storage user, identified by UID with the Onedata user. Entries for NFSv4 ACL users cannot be populated automatically, they can only be populated by local or external feeds. The table is used only in case of auto storage import enabled. Moreover, mappings for NFSv4 ACL users are used only if the Synchronize ACL option is enabled.

# Auto feed for Onedata groups table

This table cannot be populated automatically, it can only be populated by local or external feeds. The table is used only in case of auto storage import enabled with Synchronize ACL option enabled.

# Local Feed

All mappings in LUMA DB must be set explicitly by the Oneprovider admin using the REST API. It is advisable to set the mappings before inviting users to join the space, otherwise their operations on files will fail.

A comprehensive description of requests and their parameters considering the configuration of the local feed can be found in the REST API documentation (opens new window).

NOTE: For regular (non-imported) storage backends, it is enough to set the Onedata user to credentials mappings. Other mappings, if not set, are acquired the same way it is done in case of auto feed.

WARNING: In case of local feed, mappings are directly stored in the LUMA DB tables. Deleting entries from the local feed results in deleting them from the LUMA DB and vice versa.

Request Link to API
Lookup Onedata user to credentials mapping in local feed API (opens new window)
Remove Onedata user to credentials mapping in local feed API (opens new window)
Insert Onedata user to credentials mapping into local feed API (opens new window)
Modify Onedata user to credentials mapping in local feed API (opens new window)
Lookup default display credentials in local feed API (opens new window)
Remove default display credentials in local feed API (opens new window)
Insert default display credentials into local feed API (opens new window)
Lookup default POSIX credentials in local feed API (opens new window)
Remove default POSIX credentials in local feed API (opens new window)
Insert default POSIX credentials into local feed API (opens new window)
Lookup mapping of UID in local feed API (opens new window)
Remove mapping of UID in local feed API (opens new window)
Insert mapping of UID into local feed API (opens new window)
Lookup mapping of ACL user in local feed API (opens new window)
Remove mapping of ACL user in local feed API (opens new window)
Insert mapping of ACL user into local feed API (opens new window)
Lookup mapping of ACL group in local feed API (opens new window)
Remove mapping of ACL group in local feed API (opens new window)
Insert mapping of ACL group into local feed API (opens new window)

# External Feed

The LUMA DB is populated by lazily querying an external HTTP server that implements a standardized API. The space admin is responsible for the implementation and maintenance of the server.

NOTE: It is compulsory to configure the URL of external feed — see the Configuration section.

Below is the full API specification of the external LUMA feed server.

NOTE: For regular (non-imported) storage backends, it is enough to implement callback for acquiring Onedata user to credentials mappings. Other mappings, if not set, are acquired the same way it is done in case of auto feed.

# Endpoint for mapping Onedata user to storage credentials

  • URL: /storage_access/all/onedata_user_to_credentials
  • HTTP method: POST
  • accepted body:
{
    "storageId": String,         // guaranteed
    "onedataUserId": String,     // guaranteed,
    "idpIdentities": [{          // guaranteed
        "idp": String,
        "subjectId": String,
    }],
    "additionalUserDetails": {   // best effort, can be empty
        "id": String,
        "username": String,
        "emails": [String],
        "linkedAccounts": [
            "idp": String,
            "subjectId": String,
            "username": String,
            "emails": [String],
            "custom": JSON
        ]
    }
}

{
    "storageCredentials": StorageCredentials, // required
    "displayUid": Integer                     // optional
}

For more information on mapping users from an external IdP to Onedata users, see here.

# Endpoint for setting default POSIX storage credentials in the space

  • URL: /storage_access/posix_compatible/default_credentials
  • HTTP method: POST
  • accepted body:
{
    "storageId": String,    // guaranteed
    "spaceId": String       // guaranteed
}

{
    "uid": Integer,         // optional
    "gid": Integer          // optional
}

# Endpoint for setting default space display credentials in the space

  • URL: /display_credentials/default
  • HTTP method: POST
  • accepted body:
{
    "storageId": String,    // guaranteed
    "spaceId": String       // guaranteed
}

{
    "uid": Integer,          // optional
    "gid": Integer           // optional
}

# Endpoint for mapping storage UID to Onedata user

  • URL: /storage_import/posix_compatible/uid_to_onedata_user
  • HTTP method: POST
  • accepted body:
{
    "storageId": String,    // guaranteed
    "uid": Integer          // guaranteed
}

{
    "mappingScheme": "onedataUser" | "idpUser", // required
    // fields for "onedataUser" scheme
    "onedataUserId": String,                    // required if mappingScheme == "onedataUser"
    // fields for "idpUser" scheme
    "idp": String,                              // required if mappingScheme == "idpUser"
    "subjectId": String                         // required if mappingScheme == "idpUser"
}

# Endpoint for mapping NFSv4 ACL user principal to Onedata user

  • URL: /storage_import/posix_compatible/acl_user_to_onedata_user
  • HTTP method: POST
  • accepted body:
{
    "storageId": String,    // guaranteed
    "aclUser": String       // guaranteed
}

{
    "mappingScheme": "onedataUser" | "idpUser", // required
    // fields for "onedataUser" scheme
    "onedataUserId": String,                    // required if mappingScheme == "onedataUser"
    // fields for "idpUser" scheme
    "idp": String,                              // required if mappingScheme == "idpUser"
    "subjectId": String                         // required if mappingScheme == "idpUser"
}

# Endpoint for mapping NFSv4 ACL group principal to Onedata group

  • URL: /storage_import/posix_compatible/acl_group_to_onedata_group
  • HTTP method: POST
  • accepted body:
{
    "storageId": String,    // guaranteed
    "aclGroup": String      // guaranteed
}

{
    "mappingScheme": "onedataGroup" | "idpEntitlement", // required
    // fields for "onedataGroup" scheme
"onedataGroupId": String,                               // required if mappingScheme == "onedataGroup"
    // fields for "idpEntitlement" scheme
    "idp": String,                                      // required if mappingScheme == "idpEntitlement"
    "idpEntitlement": String                            // required if mappingScheme == "idpEntitlement"
}

# REST API

LUMA DB exposes a REST API (opens new window) which allows retrieving its configuration and performing lookup/delete operations in the database. Links to a comprehensive description of requests are presented in the below table:

NOTE: Deleted mapping from LUMA DB will be acquired from the currently set feed when needed. Please remember that deleting an entry from the LUMA DB in case of local feed results also in deleting it from the feed.

Request Link to API
Get LUMA DB configuration API (opens new window)
Clear LUMA DB API (opens new window)
Lookup default display credentials API (opens new window)
Remove default display credentials API (opens new window)
Lookup Onedata user to credentials mapping API (opens new window)
Remove Onedata user to credentials mapping API (opens new window)
Lookup default POSIX credentials API (opens new window)
Remove default POSIX credentials API (opens new window)
Lookup mapping of UID API (opens new window)
Remove mapping of UID API (opens new window)
Lookup mapping of ACL user API (opens new window)
Remove mapping of ACL user API (opens new window)
Lookup mapping of ACL group API (opens new window)
Remove mapping of ACL group API (opens new window)

← File popularity →