Administering Onedata REST API from command line

Overview

This tutorial describes how to manage Onedata deployment using Onezone, Oneprovider and Onepanel REST API's from command line.

Onepanel is an administration component of all Onedata services, and is automatically deployed on each node where either Onezone or Oneprovider is deployed, typically on port 9443. Onezone and Oneprovider REST API's are available on port 443.

Prerequisites

This tutorial is based on a Docker image with a preconfigured Zsh environment for accessing Onedata services, including Onedata command line clients, CDMI client and jq tool for parsing and formatting JSON. To start this environment simply execute:

docker run -it onedata/rest-cli

Before proceeding first set the following environment variables, for instance:

export ONEPANEL_HOST=https://<ONEZONE IP ADDRESS>:9443
export ONEZONE_HOST=https://<ONEZONE IP ADDRESS>
export ONEPROVIDER_HOST=https://<ONEPROVIDER IP ADDRESS>
export ONEPANEL_BASIC_AUTH=admin:password

Alternatively the environment variables can be provided in to the Docker command in a more concise form:

docker run -e ONEDATA_ZONE=<ONEZONE_IP> -e ONEDATA_PROVIDER=<ONEPROVIDER_IP> -it onedata/rest-cli

The Docker provides 3 command line utilities for each of the main Onedata services:

  • onezone-rest-cli - for accessing Onezone REST API
  • oneprovider-rest-cli - for accessing Oneprovider REST API
  • onepanel-rest-cli - for accessing Onepanel REST API

In this tutorial we are most interested in onepanel-rest-cli.

If you're connecting to a Onedata service which does not have a trusted certificate remember to add -k option to each call before the operation name.

By default Docker is configured for the latest Onedata release, but it also contains clients for all previous releases, to switch to a specific release (for instance 18.02.0) use:

onedata-select-version 18.02.0

Authentication

Onepanel supports only basic authentication, thus it is necessary to provide an administrator username and password for Onepanel either using ONEPANEL_BASIC_AUTH environment variable or by passing -u cURL option to each onepanel-rest-cli call.

User management

Onepanel allows the administrator to create, modify and remove users which have access to the Onepanel and Onezone service. This way administrators can manually add users accounts based on Basic Authentication mechanism.

onepanel-rest-cli addUser username==alice password==secret userRole==regular

Please note, that addUser operation takes on the input a JSON with username, secret and userRole, which is automatically generated by onepanel-rest-cli script for attributes passed using == sign. The above is equivalent to the following command, which takes the request body content from stdin:

echo '{"username": "alice", "password": "secret", "userRole": "regular"}' | onepanel-rest-cli addUser -

The userRole allows to create normal users when equal to regular but allows also to create additional administrator users by setting userRole to admin.

Then we can check that the user account was created properly by using Onezone REST API:

onezone-rest-cli -u alice:secret getCurrentUser | jq .

{
   "userId" : "HVR2bLYDzl2jA1eCboJcdC5oNeY8xjlfCsI58sdshEA",
   "fullName" : "Alice",
   "username" : "alice",
   "emails" : [],
   "linkedAccounts" : []
}

In order to modify user password the following command should be invoked:

onepanel-rest-cli -u admin:password modifyUser username=alice password==newsecret

In order to remove an existing account use the following command:

onepanel-rest-cli removeUser username=alice

Storage management

Each Oneprovider service can have connected multiple storage backends at the same time. Using Onepanel REST API it is possible to add storage resources to a running Oneprovider instance.

First it is necessary to select Onepanel service on Oneprovider host:

export ONEPANEL_HOST=https://<ONEPROVIDER IP ADDRESS>:9443

Now it is possible to list the available storages, which will return a list of storage Id's attached to this Oneprovider:

onepanel-rest-cli getStorages | jq .

{
  "ids": [
    "sF6AGzSiGVNSRGvxiA6p3287LUZS-nyMhDzKubrGsSM",
    "Hx0HFUdKiDFgMdNDuFv7uwvpr5dew5VHS2N2kYa9haQ"
  ]
}

Now to get details of a specific storage we can use the following command:

onepanel-rest-cli -k getStorageDetails id=sF6AGzSiGVNSRGvxiA6p3287LUZS-nyMhDzKubrGsSM | jq .
{
  "id": "sF6AGzSiGVNSRGvxiA6p3287LUZS-nyMhDzKubrGsSM",
  "name": "NFS",
  "type": "posix",
  "readonly": false,
  "insecure": false,
  "lumaEnabled": true,
  "lumaUrl": "http://192.168.1.200:8080/api/v3/luma",
  "lumaCacheTimeout": 1,
  "mountPoint": "/mnt/nfs"
}

In order to add new storage, for example an S3 service instance, first create the storage configuration in a temporary file s3.json, for example:

{
  "S3Test" : {
    "type" : "s3",
    "hostname" : "192.168.1.200:8000",
    "bucketName" : "test1",
    "accessKey" : "accessKey1",
    "secretKey" : "verySecretKey1",
    "insecure": true,
    "lumaEnabled": false
  }
}

and then simply pass the contents of the s3.json to the addStorage operation of onepanel-rest-cli:

cat s3.json | onepanel-rest-cli addStorage -

Now we can check again, if the storage was added successfully:

onepanel-rest-cli getStorageDetails id=TCRg-YIpyXBdh55pXQCERF8YjSZcmMxXxHmWIVJk9wM | jq .

{
   "NFS" : {
    "id": "sF6AGzSiGVNSRGvxiA6p3287LUZS-nyMhDzKubrGsSM",
    "name": "NFS",
    "type": "posix",
    "readonly": false,
    "insecure": false,
    "lumaEnabled": true,
    "lumaUrl": "http://192.168.1.200:8080/api/v3/luma",
    "lumaCacheTimeout": 1,
    "mountPoint": "/mnt/nfs"
  },
   "S3Test" : {
      "type" : "s3",
      "bucketName" : "test1",
      "scheme" : "http",
      "hostname" : "192.168.1.200:8000",
      "id" : "eD8jiGHITXuM4o7gfGc5c2_gVOxRYF1vP4O_hiwlsTE"
   }
}

Space support

One of the most frequent administrative tasks is supporting user spaces based on tokens passed to administrators by users. In order to support a space on a selected storage with a specific quota, first it is necessary to select Onepanel on Oneprovider host:

export ONEPANEL_HOST=https://<ONEPROVIDER IP ADDRESS>:9443
onepanel-rest-cli supportSpace token==MDAxNmxvY2F00aW9uIHJlZ2lzdHJ5CjAwM2JpZGVudGlmaWVyIDI2THNTM3RkdGNoZ00pUS3ZMb3JkMFQ00cDZPSXhWQnVvZHAwYUxNWHVxdzAKMDAyOGNpZCB00b2tlblR5cGUgPSBzcGFjZV9zdXBwb3J00X3Rva2VuCjAwMmZzaWduYXR1cmUg1EKnr7dPbh00I01X02wx8ULLjNt02HzBtfMxTp3jtse01vFsK size:=1073741824 storageName==NFS | jq '.'

{
   "id" : "gle-Xf89_TX77x1RITRkfmhII7aCfNYS0VpiCq93h-M"
}

We can now check that the space has been supported by checking the list of spaces in the Oneprovider interface (as administrator):

oneprovider-rest-cli -u admin:password getAllSpaces | jq '.'
[
  {
    "spaceId": "YCGwWg-b7lCGNuhQz5ufHrrxX8Zu0pCFgkgZQ3y-WO0",
    "name": "Personal files"
  }
]

It is also possible to revoke the support for a specific space by using a space GUID:

onepanel-rest-cli revokeSpaceSupport id=gle-Xf89_TX77x1RITRkfmhII7aCfNYS0VpiCq93h-M

Please note that revoking space support may result in data loss in case some of the files were only replicated on this Oneprovider instance.