REST API in Onedata

REST API in Onedata can be used to access all of its main components and perform operations as shown in the diagram below:

REST API in Onedata

To access Onedata through REST API you need to have an access token - by now you should have created them in one of the previous sessions (you can find them in the Tokens tab). Before we start, export the following environment variables:

export PROVIDER_ACCESS_TOKEN=...
export ZONE_ACCESS_TOKEN=...
export PANEL_ACCESS_TOKEN=... # access token to Oneprovider Panel
export ZONE_HOST=demo.onedata.org
export PROVIDER_HOST=... # domain of your Oneprovider
export USER_DIR=... # name of your subdirectory in alpha-11p
export SPACE_ID=... # ID of space alpha-11p, copy from GUI using the "three dots" next to the space

Privileges management using REST API

Privileges in Onedata resources can be managed using the API. As privileges management (as part of space management in the case of this exercise) is performed in Onezone we will use $ONEZONE_ACCESS_TOKEN and Onezone REST API.

1. With the following command, you can get information about your user:

   curl -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X GET https://$ZONE_HOST/api/v3/onezone/user | jq
   

   Save the user ID from the response under the USER_ID variable:

   export USER_ID=...
   

Privileges management using REST API

2. Create a group belonging to space alpha-11p:

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X POST https://$ZONE_HOST/api/v3/onezone/spaces/$SPACE_ID/groups \
   -H "Content-Type: application/json" -d '{
     "name": "some-group"
   }'
   

   URI of the added group in form https://$ZONE_HOST/api/onezone/v3/spaces/{space_id}/groups/{group_id} is returned in the response Location header. Save the group ID under the GROUP_ID variable:

   export GROUP_ID=...
   

Privileges management using REST API

3. You can check what privileges the group has in the space alpha-11p:

   curl -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X GET \
   https://$ZONE_HOST/api/v3/onezone/spaces/$SPACE_ID/groups/$GROUP_ID/privileges | jq
   

4. With the use of REST API privileges can be easily changed - you will now remove SpaceView privilege for a group:

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -H "Content-type: application/json" -X PATCH \
   -d '{"revoke": ["space_view"]}' https://$ZONE_HOST/api/v3/onezone/spaces/$SPACE_ID/groups/$GROUP_ID/privileges
   

5. Adding privileges works the same - simply change revoke to grant (NOTE: you can combine revoke and grant in one command when changing multiple privileges).

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -H "Content-type: application/json" -X PATCH \
   -d '{"grant": ["space_view"]}' https://$ZONE_HOST/api/v3/onezone/spaces/$SPACE_ID/groups/$GROUP_ID/privileges
   

All other Onedata resources (i.e. groups, clusters, harvesters, etc) can have privileges modified similarly.

File writing using REST API

As data management is performed in Oneprovider, we will use $PROVIDER_ACCESS_TOKEN and Oneprovider REST API.

0. (Paris only) create a file in space alpha-11p:

   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" \
   -X PUT "https://$PROVIDER_HOST/api/v3/oneprovider/data/$SPACE_ID/path/write_rest_api_exercise"
   

   Send the created file ID to all other participants.

1. Save the received file ID under the variable WRITE_FILE_ID:

   export WRITE_FILE_ID=...
   

File writing using REST API

2. Write data to file at specified offset (save following value under OFFSET variable):

   Paris	Marseille	Lyon	Toulouse	Nice	Nantes	Montpellier	Strasbourg	Bordeaux	Lille	Rennes
   0	8	16	24	32	40	48	56	64	72	80

   export OFFSET=...
   

   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" \
   -X PUT "https://$PROVIDER_HOST/api/v3/oneprovider/data/$WRITE_FILE_ID/content?offset=$OFFSET" \
   -H "Content-Type: application/octet-stream" -d "01234567"
   

3. In GUI see the file’s data distribution (right click on the file -> Data distribution)

Manual import

Manual import allows registering specific files lying on a storage backend into a Onedata space. For more details, see the documentation here.

In this exercise, you create a space supported by a manually imported storage, using REST API.

Create a new space - as space management is done in Onezone we will use Onezone REST API:

curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -H "Content-type: application/json" \
-X POST -d '{ "name" : "manual_import_space" }' https://$ZONE_HOST/api/v3/onezone/user/spaces

In the response, you should receive a new space URI in the form of https://$ZONE_HOST/api/onezone/v3/user/spaces/{id}.

Save the space ID from this response in the MANUAL_IMPORT_SPACE_ID variable:

export MANUAL_IMPORT_SPACE_ID=...

Manual import

Create a new storage backend of the type HTTP - this storage backend type can be any HTTP or HTTPS compatible server. It is supported only with readonly option enabled and is dedicated to manual import mode.

Storage management is done in Oneprovider panel - so we will use $PANEL_ACCESS_TOKEN and Onepanel REST API.

curl -v -H "X-Auth-Token: $PANEL_ACCESS_TOKEN" -X POST https://$PROVIDER_HOST/api/v3/onepanel/provider/storages \
-H "Content-Type: application/json" -d '{
    "manual_import_http": {
        "type": "http",
        "importedStorage": true,
        "readonly": true,
        "endpoint": "https://packages.onedata.org/"
    }
}'

Save the created storage ID in the variable HTTP_STORAGE_ID:

export HTTP_STORAGE_ID=...

Manual import

Now let's support the newly created space with this storage backend. First, you need to create a support token:

curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X POST https://$ZONE_HOST/api/v3/onezone/user/tokens/named \
-H "Content-type: application/json" -d '{
  "name": "manual-import-space-support",
  "type": {
    "inviteToken": {
      "inviteType": "supportSpace",
      "spaceId": "'$MANUAL_IMPORT_SPACE_ID'"
    }
  }
}'

Save it under the variable SUPPORT_TOKEN:

export SUPPORT_TOKEN=...

Manual import

Now you can support the space:

curl -v -H "X-Auth-Token: $PANEL_ACCESS_TOKEN" -X POST https://$PROVIDER_HOST/api/v3/onepanel/provider/spaces \
-H "Content-Type: application/json" -d '{
    "token": "'$SUPPORT_TOKEN'",
    "size": "10000000",
    "storageId": "'$HTTP_STORAGE_ID'",
    "storageImport": {
        "mode": "manual"
    }
}'

You are ready to register a file using manual storage import:

curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" -X POST "https://$PROVIDER_HOST/api/v3/oneprovider/data/register" \
-H 'Content-Type: application/json' -d '{
  "spaceId": "'$MANUAL_IMPORT_SPACE_ID'",
  "storageId": "'$HTTP_STORAGE_ID'",
  "storageFileId": "https://packages.onedata.org/apt/ubuntu/2002/dists/focal/Release",
  "destinationPath": "/manually_imported_file"
}'

You can confirm in GUI that the new space with the file is visible.

File API tab

In this exercise, you will learn about the file API tab in GUI. This tab is available for any file in Oneprovider and can be used to quickly access a file with REST API.

1. In GUI go to the alpha-11p space and enter your directory
2. Right click on the FirstDataset directory and select Information.
3. In the modal select Api tab - here you can see prepared REST API commands for various file related operations.
4. Select Get RDF metadata operation.
5. Copy the command, replace variable TOKEN with PROVIDER_ACCESS_TOKEN and run it

A similar tab can be found for spaces - to access it click on three dots next to the space name and select REST API.

Shared file API tab

In this exercise, you will learn about the API tab in a public share view.

1. In GUI go to the Shares, Public Data tab and select any share, go to public view.
2. Right click on any file, select Information and go to the Api tab.
3. Here you can also see prepared REST API commands for the file related operations, however list of available operations is shorter, as not all operations are available in share mode.
4. Select any command and run it - notice that you do not need any access token - this is share mode, which is publicly available to everyone.

Harvesters query with REST API

In the following exercise, you will learn how to use REST API to perform data discovery queries.

1. In GUI go to one of your harvesters created in previous sessions (the Discovery tab).
2. In the top right corner click on the REST API button - the command in the modal will perform the same query you see in GUI. (use ZONE_ACCESS_TOKEN as TOKEN in generated command).
3. Play around with different queries and see how the command changes.

Transfers and QoS REST API

1. Create a file:

   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" \
   -X PUT "https://$PROVIDER_HOST/api/v3/oneprovider/data/$SPACE_ID/path/$USER_DIR/rest_api_exercise" \
   -H "Content-Type: application/octet-stream" -d "0123456789"
   

   Save the created file ID under the FILE_ID variable:

   export FILE_ID=...
   

2. Replicate the file to any other Oneprovider (save its ID under PROVIDER_ID variable; you can find it in the Providers tab in GUI):

   export PROVIDER_ID=...
   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" -X POST "https://$PROVIDER_HOST/api/v3/oneprovider/transfers" \
   -H "Content-Type: application/json" -d '{
       "type": "replication",
       "replicatingProviderId": "'$PROVIDER_ID'",
       "dataSourceType": "file",
       "fileId": "'$FILE_ID'"
   }'
   

Transfers and QoS REST API

3. See in GUI in distribution (right click in file -> Data distribution) that the file is replicated on 2 Oneproviders.

4. Add QoS requirement to require this file on any 3 storage backends:

   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" -X POST "https://$PROVIDER_HOST/api/v3/oneprovider/qos_requirements" \
   -H "Content-Type: application/json" -d '{
       "expression": "anyStorage",
       "replicasNum": 3,
       "fileId": "'$FILE_ID'"
   }'
   

5. See in GUI in distribution (right click on the file -> Data distribution) that the file is replicated on at least 3 Oneproviders.

Datasets and Archives REST API exercise

1. Establish a dataset on the file from the previous exercise:

   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" -X POST "https://$PROVIDER_HOST/api/v3/oneprovider/datasets" \
   -H "Content-Type: application/json" -d '{"rootFileId": "'$FILE_ID'"}'
   

   Save the created dataset ID under the DATASET_ID variable:

   export DATASET_ID=...
   

2. Now create an archive from this dataset:

   curl -v -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" -X POST "https://$PROVIDER_HOST/api/v3/oneprovider/archives" \
   -H "Content-Type: application/json" -d '{
       "datasetId": "'$DATASET_ID'",
       "config": {
           "includeDip": false,
           "layout": "plain"
       },
       "description": "Dataset archived with the use of REST API."
   }'
   

   You can confirm in GUI that the dataset and archive were created.

Tokens REST API - temporary tokens

The following exercise will show how to manage tokens using the REST API in Onedata. For more details consult the documentation.

1. Create a temporary token valid for one hour. Temporary tokens must be confined with a time caveat and are not persisted (cannot be retrieved, listed, revoked, updated or deleted). However, all temporary tokens for a user can be revoked.

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X POST \
   -d '{"type": {"accessToken":{}}, "caveats": [{"type": "time", "validUntil": '$(($(date +%s) + 3600))'}]}' \
   -H 'Content-type: application/json' \
   https://$ZONE_HOST/api/v3/onezone/user/tokens/temporary
   

   Save it under the TEMPORARY_TOKEN variable.

   export TEMPORARY_TOKEN=...
   

   For more details about temporary tokens see the documentation.

Tokens REST API - temporary tokens

2. Get information about the file using the generated token:

   curl -H "X-Auth-Token: $TEMPORARY_TOKEN" \
   -X GET "https://$PROVIDER_HOST/api/v3/oneprovider/data/$FILE_ID" | jq
   

3. You can see the details of your token with the examine token request:

   curl -d '{"token": "'$TEMPORARY_TOKEN'"}' -H 'Content-type: application/json' \
   https://$ZONE_HOST/api/v3/onezone/tokens/examine | jq
   

4. Now confine this token with additional caveats (the below command will confine the token with data readonly caveat):

   curl -d '{
     "token": "'$TEMPORARY_TOKEN'",
     "caveats": [
       {"type": "data.readonly"}
     ]
   }' \
   -H 'Content-type: application/json' \
   https://$ZONE_HOST/api/v3/onezone/tokens/confine | jq
   

Tokens REST API - temporary tokens

5. Send this confined token to your colleague sitting to your left.

6. Save received from colleague token under RECEIVED_TOKEN:

   export RECEIVED_TOKEN=...
   

7. Now try to write to a file - it will fail because the token allows only for readonly operations.

   curl -v -H "X-Auth-Token: $RECEIVED_TOKEN" \
   -X PUT "https://$PROVIDER_HOST/api/v3/oneprovider/data/$FILE_ID/content" \
   -H "Content-Type: application/octet-stream" -d "0123456789"
   

Tokens REST API - temporary tokens

8. Now let's add a new caveat to the already confined token, this time whitelisting an IP address (for the purpose of this exercise let's add an IP address, that is definitely incorrect):

   curl -d '{
     "token": "'$RECEIVED_TOKEN'",
     "caveats": [
       {"type": "ip", "whitelist": ["8.8.8.8"]}
     ]
   }' \
   -H 'Content-type: application/json' \
   https://$ZONE_HOST/api/v3/onezone/tokens/confine | jq
   

   Save it under the IP_CONFINED_TOKEN variable:

   export IP_CONFINED_TOKEN=...
   

   Once added a token's caveat cannot be removed. All available token caveats are described in the documentation.

Tokens REST API - temporary tokens

9. Any request done with such a token will now fail, you may try listing files in a space:

   curl -H "X-Auth-Token: $IP_CONFINED_TOKEN" \
   -X GET "https://$PROVIDER_HOST/api/v3/oneprovider/data/$SPACE_ID/children" | jq
   

10. Revoke all your temporary tokens - any operation with temporary tokens on behalf of your user will now fail:

    curl -v -H "x-auth-token: $ZONE_ACCESS_TOKEN" -X DELETE \
    https://$ZONE_HOST/api/v3/onezone/users/$USER_ID/tokens/temporary
    

11. Compare the results of the same request (getting information about a file) using TEMPORARY_TOKEN, RECEIVED_TOKEN (after your colleague revoked his temporary tokens) and PROVIDER_ACCESS_TOKEN tokens - only the last one will work, as it is not a temporary token:

    curl -H "X-Auth-Token: $TEMPORARY_TOKEN" -X GET "https://$PROVIDER_HOST/api/v3/oneprovider/data/$FILE_ID"
    curl -H "X-Auth-Token: $RECEIVED_TOKEN" -X GET "https://$PROVIDER_HOST/api/v3/oneprovider/data/$FILE_ID"
    curl -H "X-Auth-Token: $PROVIDER_ACCESS_TOKEN" -X GET "https://$PROVIDER_HOST/api/v3/oneprovider/data/$FILE_ID" | jq
    

Tokens REST API - named tokens

The following exercise will show how to revoke, unrevoke, and delete named tokens and how to use the consumer caveat.

1. Create a new named access token:

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X POST https://$ZONE_HOST/api/v3/onezone/user/tokens/named \
   -H "Content-type: application/json" -d '{
     "name": "named-access-token",
     "type": {"accessToken": {}}
   }' | jq
   

   Save the created token and token ID under the NAMED_TOKEN and TOKEN_ID variables respectively.

   export NAMED_TOKEN=...
   export TOKEN_ID=...
   

Tokens REST API - named tokens

2. Now confine it with the consumer caveat (let's limit the token only to members of the group created in one of the previous exercises):

   curl -d '{
       "token": "'$NAMED_TOKEN'",
       "caveats": [
           {"type": "consumer", "whitelist": ["grp-'$GROUP_ID'"]}
       ]
   }' \
   -H 'Content-type: application/json' \
   https://$ZONE_HOST/api/v3/onezone/tokens/confine | jq
   

   Save the created token under the CONFINED_NAMED_TOKEN variable:

   export CONFINED_NAMED_TOKEN=...
   

   For more details about consumer caveats see the documentation.

Tokens REST API - named tokens

3. Perform any request (like listing your groups) with both tokens (NAMED_TOKEN and CONFINED_NAMED_TOKEN):

   curl -H "X-Auth-Token: $NAMED_TOKEN" -X GET https://$ZONE_HOST/api/v3/onezone/user/groups | jq
   

   curl -H "X-Auth-Token: $CONFINED_TOKEN" -X GET https://$ZONE_HOST/api/v3/onezone/user/groups
   

   The first request succeeds, however, the second one fails - that is because a token confined with a consumer caveat must be accompanied by an identity token sent under the x-onedata-consumer-token header.

   For more details about identity tokens see the documentation.

Tokens REST API - named tokens

4. **Take your identity token from the GUI, or create a new one using the API:

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X POST https://$ZONE_HOST/api/v3/onezone/user/tokens/named \
   -H "Content-type: application/json" -d '{
     "name": "identity-token",
     "type": {"identityToken": {}}
   }' | jq
   

   Save the created token under the IDENTITY_TOKEN variable:

   export IDENTITY_TOKEN=...
   

5. Now try again performing the request, this time providing the required identity token:

   curl -H "X-Auth-Token: $CONFINED_NAMED_TOKEN" -H "x-onedata-consumer-token: $IDENTITY_TOKEN" \
   -X GET https://$ZONE_HOST/api/v3/onezone/user/groups | jq
   

   This time the request succeeds as you are a member of the group, token is limited to.

Tokens REST API - named tokens

6. Named tokens can be revoked - after that a token, and all tokens created from it by adding new caveats, will no longer be accepted:

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -H "Content-type: application/json" \
   -X PATCH -d '{"revoked": true}' \
   https://$ZONE_HOST/api/v3/onezone/tokens/named/$TOKEN_ID
   

7. Try to perform a request with both NAMED_TOKEN and CONFINED_NAMED_TOKEN:

   curl -H "X-Auth-Token: $NAMED_TOKEN" -X GET https://$ZONE_HOST/api/v3/onezone/user/groups
   

   curl -H "X-Auth-Token: $CONFINED_NAMED_TOKEN" -H "x-onedata-consumer-token: $IDENTITY_TOKEN" \
   -X GET https://$ZONE_HOST/api/v3/onezone/user/groups
   

   Both requests fail because the NAMED_TOKEN token has been revoked.

Tokens REST API - named tokens

8. Revoked named tokens can be unrevoked - it is done similarly to revoking:

   curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -H "Content-type: application/json" \
   -X PATCH -d '{"revoked": false}' \
   https://$ZONE_HOST/api/v3/onezone/tokens/named/$TOKEN_ID
   

9. Try to perform a request again with both NAMED_TOKEN and CONFINED_NAMED_TOKEN:

   curl -H "X-Auth-Token: $NAMED_TOKEN" -X GET https://$ZONE_HOST/api/v3/onezone/user/groups | jq
   

   curl -H "X-Auth-Token: $CONFINED_NAMED_TOKEN" -H "x-onedata-consumer-token: $IDENTITY_TOKEN" \
   -X GET https://$ZONE_HOST/api/v3/onezone/user/groups | jq
   

   Now both requests succeed because the NAMED_TOKEN token has been unrevoked.

Tokens REST API - named tokens

10. Named tokens can also be deleted - similarly to revocation, after deletion a token, and all tokens created from it by adding new caveats, will no longer be accepted. However, deletion is not reversible.

    curl -v -H "X-Auth-Token: $ZONE_ACCESS_TOKEN" -X DELETE \
    https://$ZONE_HOST/api/v3/onezone/tokens/named/$TOKEN_ID
    

11. Try to perform a request again with both NAMED_TOKEN and CONFINED_NAMED_TOKEN:

    curl -H "X-Auth-Token: $NAMED_TOKEN" -X GET https://$ZONE_HOST/api/v3/onezone/user/groups
    

    curl -H "X-Auth-Token: $CONFINED_NAMED_TOKEN" -H "x-onedata-consumer-token: $IDENTITY_TOKEN" \
    -X GET https://$ZONE_HOST/api/v3/onezone/user/groups
    

    Both requests fail because the NAMED_TOKEN token has been deleted.