Logs

In the context of a Docker-compose-based deployment, all logs related to Docker and applications are easily accessible using the docker-compose logs command invoked inside the Onedatify directory. Additionally, application logs are persisted and can be found in the persistence directory:

oneprovider_conf/
└── var
    └── log
        ├── cluster_manager/
        ├── oneprovider/
        ├── op_panel/
        └── op_worker/

Among many types of logs, the ones based on syslog logging levels (debug, info, error, etc.) are the most relevant ones for administrators. For more details, consult the documentation.

Take a look at the op_worker service logs:

~$ tail -n 100 /opt/onedata/onedatify/oneprovider_conf/var/log/op_worker/info.log

Backup

Backing up the service boils down to creating snapshots of the Onedatify directory which contains the persistence directory with database files and service configuration. For more details, consult the documentation.


NOTE: Administrators should also back up the underlying storage systems that store the physical data exposed via logical Onedata spaces. This guide does not cover storage data backups, you should use backup procedures recommended for specific storage backend types.

Offline backups

Offline backups guarantee the complete integrity of the snapshot but require a certain downtime of the service.

  1. Stop the service.

    ~$ onedatify stop

  2. Copy the installation directory to a safe place.

    ~$ sudo mkdir -p /opt/onedata/oneprovider-backups
~$ sudo rsync -as /opt/onedata/onedatify /opt/onedata/oneprovider-backups/$(date +%Y.%m.%d)

  3. Start the service.

    ~$ onedatify start

NOTE: In multinode deployments, the procedure must be done on each host and synchronized, i.e. all nodes should be stopped, then backed up and started together.

Live backups

In this procedure, the service nodes are not stopped, but the snapshot is performed while the service is running. The procedure embraces the LVM volume that has been created for the persistence directory.

Despite using LVM's atomic snapshots, this approach does not guarantee the complete integrity of the backups, as in rare cases the backup may happen when the application's state is not fully flushed from memory to the disk. In such a case, changes from the last couple of seconds will be lost, but if we consider restoring the service from a backup, changes that happened since the snapshot will be lost anyway.

Doing a live backup

  1. Create an LVM snapshot.

    ~$ sudo lvcreate -s -n lvol0-snap -l10%ORIGIN /dev/onedata-vg/lvol0

  2. Mount the snapshot.

    ~$ sudo mkdir -p /snapshots/opt/onedata
~$ sudo mount /dev/onedata-vg/lvol0-snap /snapshots/opt/onedata/

  3. Inspect the mounted snapshot.

    ~$ ls /snapshots/opt/onedata/

  1. At this point, in a production setup, you should upload the backup to a different location (at least one) to minimize the risk of data loss in case of a significant data center failure.

    The script odbackup.sh from the onedata-deployments repository can be used for that purpose. The script uploads the backups to an S3 bucket using s3cmd, which needs to be configured before running the script.

    It's enough to create a cron job to run the backup script daily (or as often as you need) and you will thank yourself later! (but hopefully not)

  2. Clean up.

    ~$ sudo umount /dev/onedata-vg/lvol0-snap
~$ sudo lvremove -y /dev/onedata-vg/lvol0-snap

Upgrade

Let's upgrade the Oneprovider service to the edge channel release:

  1. Find out the current build version of the Oneprovider service:

    ~$ curl -k --silent https://127.0.0.1/api/v3/oneprovider/configuration | jq .build

  2. Pull the docker image to make the upgrade smoother:

    ~$ docker pull onedata/oneprovider:21.02.3-develop

  3. Run the upgrade command. The service will be stopped, backed up, and started on a newer docker image.

    ~$ onedatify upgrade -v onedata/oneprovider:21.02.3-develop

  4. Inspect the created backup:

    ~$ ls -p /opt/onedata/oneprovider-backups

  5. Once again check the build version and see that it has changed.

Upgrade process anatomy

Any required upgrade procedures are implemented internally in the software. When a new version is started on top of pre-existing persistence data originating from an older version, suitable procedures are run. The procedures may be long-lasting but are idempotent and resistant to service restarts.

Oneprovider configuration panel (Onepanel)

The Onepanel service runs on every node of a Oneprovider cluster and offers a Web GUI and REST API for administering the cluster. For more details, consult the documentation.

The Web GUI is integrated within the CLUSTERS tab in the navigation bar.

Overview

In the Overview tab you can see a short summary of the Oneprovider cluster — its basic details, as well as the total number of granted space support and the number of configured storage backends.

screenshot centered

Nodes

In order to provide failover and high availability of Onedata services, Oneprovider can consist of multiple nodes (a cluster). For the needs of this workshop, we are deploying simple one-noded clusters.

The Nodes view presents a list of all cluster nodes, with information about which internal services run on the nodes. In the second tab, nodes' IP addresses can be configured — this is especially important when subdomain delegation is used, as in our case.

screenshot

DNS setup

Diagnostic tools that help you ensure that the cluster is reachable (its domain is properly resolved in DNS, pointing to the cluster node IP addresses).

The status of the last check should be OK — let us know if it's not.

For more details, consult the documentation.

screenshot

Web certificate

Visit this tab to view details about the Web certificate used for the Oneprovider HTTPS servers. The certificate has been automatically obtained using Let's Encrypt during the deployment process.

screenshot centered

Members

Here you can view and manage members of your cluster, in an analogous way as practiced during one of the previous chapters. Users and groups added to the cluster gain access to it and can view and manage the cluster to the extent defined by assigned privileges. For more details, consult the documentation. screenshot centered Now, add the Workshops group as a member, so that everyone can have a look at different providers in the environment. Assign only the view privilege, we don't want you to tinker with each other's configuration.

Provider

Summary of information provided during Oneprovider registration in Onezone. Some of them may be modified here. It's also the place for the provider deregistration button. screenshot centered Do not press the red button...

Storage backends

The list of configured storage backends for this Oneprovider. Here you can add new storage backends and modify existing ones. Click to expand the default backend that was added during the deployment and examine the information. For more details, consult the documentation. screenshot centered

Spaces

The list of spaces that are supported by this Oneprovider. Here you can also support new spaces or modify the existing ones. Click to expand and examine the supported space. For more details, consult the documentation.

screenshot centered

Next chapter:

Building the Onedata@FR — practice