Datasets in a space

Open Archival Information System (OAIS)

Exercise

Establishing a dataset and creating archives

1. Navigate to the directory created for your username.
2. Open the context menu for the FirstDataset directory and choose Datasets.

3. Click on the Establish dataset in the datasets right-side panel.

4. The panel should now present a summary of the dataset established on the FirstDataset directory.

   

5. Navigate to the Archives tab in the datasets panel.
6. Click on the Create a new archive button.
7. In the Create archive modal, write a description of the archive and use the default archivisation settings.
8. Click on the Create button.
   

9. Upon successful creation, the archive list should include the newly created archive.

   Various actions for the archive are available in the context menu. Try:

   • viewing the audit log,
   • downloading the archive and viewing the contents.

   

10. Double-click on the archive row to browse through its contents.

    Note that the custom metadata set on the original files in previous exercises have also been snapshotted with the file data — use the badge to view it.

    When you finish browsing, use the breadcrumbs to navigate back to the archive list.

    

11. Use the Create archive button to create another archive with a Second archive description and default settings.

    Now, you should see two archives on the list.

    

Incremental archives

A regular archive is a full copy of the dataset contents, occupying an additional storage quota. Storage space can be saved by creating incremental archives, whenever possible.

An incremental archive is created upon the chosen base archive and reuses all files that haven't been modified between the snapshots, creating hard links to them (which take no storage space). Only the modified and new files are copied.

Incremental archives in a space

6. In a few moments, the new archive should pop up on the list.
   Note the Base archive indication.

   

   Also, note that the Owner of the archive differs from that of the archive created previously by your colleague.

7. Double-click on the archive row to browse its contents and navigate to the directory with files.

   Original, unmodified files from the first archive have 2 hard links badges, indicating that the incremental archive reuses the data stored in the base one.

   The new files do not have the badge, since they have been fully copied.

   

Archive recall

While archives serve as immutable snapshots, the dataset may evolve in time as the data in the space changes. Sometimes it is desirable to bring back a certain snapshot into the space filesystem to work on it.

To do that, users may recall the archive. The process copies the data at the chosen path in the space, creating a new file/directory that is not logically linked to the original dataset.

Exercise: recalling an archive

1. Navigate to the dataset management panel of the previously established dataset.

2. Right-click on the latest archive and choose the Recall to... action.

3. In the Recall archive panel that has appeared, navigate to the directory with your name.

5. Click on the Recall button.

6. Close the Datasets panel and navigate to the directory created for your username in the file browser. During the recall process, its progress is presented in the RecalledDataset1 row. After completion, the Recalled badge will appear.

   

7. Click on the Recalled badge.

9. Close the panel and open the recalled directory in the file browser. Examine its contents.

   The directory can be used just as any other in the space and the only difference is the context information that this is a recalled archive, indicated by the badge.

   

Dataset hierarchy and nested archives

Datasets can be established on files or directories lying inside other datasets. This way, a hierarchy of datasets can be built and then embraced to create nested archives.

Exercise: modifying and browsing dataset hierarchy

1. Navigate to the directory with the dataset established in previous exercises in the file browser.

2. Find inside the directory called Nested1. Right-click on it and choose Datasets from the context menu.

3. In the datasets panel, you should see an indication that there is no established dataset on this directory. Below that, the list of Ancestor datasets should include the previously created dataset (click to expand).

4. Click on the Establish dataset button on the right side of the panel.

   

7. Open the datasets panel for the Nested4 directory and establish a dataset.

   Now you should see the following hierarchy:

   • FirstDataset,
   • Nested1,
   • Nested4.
   
   

   Non-dataset directories on the path are not listed.

9. Using the breadcrumbs at the top navigate to the first item on the path, which is the space root directory.

   Now, you are viewing the root of the space's dataset hierarchy. You can also bring up this view from the left sidebar menu: Space > Space name > Datasets, Archives.

Nested archives

Nesting datasets allow composing structures of desired granularity. An embedded dataset can be a logical whole that's useful individually, and at the same time be a part of a bigger data collection, vital for its completeness.

Those nested structures are also important when archives are created; on the ancestor dataset level, users may choose to create a monolithic archive or create nested archives in the process. This way, a set of linked archives will be created.

Nested archives in the dataset tree

Exercise: creating nested archives

1. Navigate to the Datasets, Archives view of the space.

2. Right-click on the FirstDataset dataset on the list and choose Create archive from the context menu.

3. Create an archive with default settings — do not use the Create nested archives option just yet.

4. Browse files of the newly created archive.

   Note that Nested1 and Nested4 directories have been fully copied into the archive.

5. Create the next archive for the FirstDataset dataset. This time enable the Create nested archives option.

6. Browse files of the newly created archive.

   Note that Nested1 and Nested4 directories are symbolic links. These links point to directories copied into the nested archives that were created from those nested datasets.

   

9. Browse the archive files.

   The file tree starts from the Nested1 directory, to which the symbolic link in the FirstDataset dataset points.

   Go through the nested directories to see that there is also a symbolic link to the Nested4 directory.

   

Dataset protection flags

To prevent changes in the dataset, users can set temporary protection flags, which cause the files and directories in the dataset to be protected from, accordingly:

• Data protection — modifying their content or being deleted,
• Metadata protection — modifying their metadata, such as custom JSON/RDF/xattr metadata, permissions, or ACLs.

The protection flags are inherited by children datasets — datasets lower in the hierarchy tree have protection at least as strong as their ancestors.

Exercise: setting protection on the datasets

1. Navigate to the parent directory of the dataset from previous exercises in the web file browser and open the datasets panel using the Dataset badge.

   You should see two indicators in the upper-right corner of the panel say:

   • directory's data is write enabled,
   • directory's metadata is write enabled.

   This means that the dataset is not protected — users can modify the data and metadata of the files and directories inside.

2. Click on the toggle in the Data write protection column of the Datasets hierarchy table to enable it.

   Note that the second toggle, Metadata write protection, requires that first the data protection is enabled. Do not enable it just yet.

   The two indicators in the upper-right corner of the panel should now show:

   • Directory's data is write protected,
   • Directory's metadata is write enabled,

   which means that the users cannot modify the data but still can change the metadata of the dataset contents, e.g. edit custom JSON metadata. Note that files cannot be renamed while the parent directory's data is write-protected.

3. Close the datasets panel.

   The Dataset badge on the directory now has an additional icon that signs enabled dataset data protection.

4. Enter the dataset directory in the web GUI file browser.

   Verify that creating directories or uploading files is now disallowed, like any other data modifying operation.

5. Navigate to the Nested3 directory and open the datasets panel for the Nested4 item.

6. Expand the Ancestor datasets section in the Dataset hierarchy table.

   The summarized ancestor dataset protection is presented in the first row, while the protection set for particular ancestor datasets is shown in each row separately. You can configure all the ancestor datasets protection in this table.

7. Enable both Data write protection and Metadata write protection for the Nested1 dataset in the table.

   

8. Close the panel and enter the Nested4 directory in the file browser.

   All operations modifying data or metadata of the files are now disallowed. Verify that — for example, try to add the Basic (key-value) metadata of a file inside the Nested4 / Nested5 path. The protection has been inherited from ancestor datasets.

BagIt archive

BagIt is a set of hierarchical file system conventions for organizing and transferring digital content.

A "bag" consists of the "payload" (actual content) with "tags" serving as metadata files documenting storage and transfer details. A mandatory tag file includes a manifest listing every file and its corresponding checksums.

The term "BagIt" is inspired by the "enclose and deposit" method, also known as "bag it and tag it".