Metadata

In the Onedata system, metadata is organized into 2 levels and regards every file/directory:

• Filesystem attributes — built-in filesystem metadata such as file size, creation and modification timestamps, POSIX permissions, etc.
• Custom metadata — arbitrary, user-defined information:
  • extended attributes — a.k.a. basic attributes or xattrs; key-value pairs, compatible with POSIX extended attributes,
  • JSON document,
  • RDF document (XML).

All types of custom metadata can coexist for the same file/directory.

See the documentation here.

Exercise

Assigning custom JSON metadata to files

In this exercise, we will assign custom metadata to several graphic files. It will include the properties of the graphics they contain.

1. Go to the file browser of the space alpha-11p, and enter the directory $yourName/FirstDataset/Photos.
2. Right-click on the file artem.jpg and choose Metadata.
3. Switch to the JSON editor and enter:

   {
      "imageAuthor": "???",
      "imageCategory": "nature",
      "imageWidth": 1680,
      "imageHeight": 1200
   }
   

   but change the imageAuthor to your first name.
4. Click the Save button.

Exercise

Assigning custom RDF metadata to a directory

1. Go to the file browser of the space alpha-11p and enter your directory.
2. Right-click on the directory FirstDataset and choose Metadata.
3. Switch to the RDF editor and enter:

   <rdf:RDF
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:si="https://www.mypage.fake/rdf/">
   
      <rdf:Description rdf:about="directory">
         <si:title>Dataset with a choice of photos</si:title>
         <si:author>Various</si:author>
      </rdf:Description>
   
   </rdf:RDF>
   

4. Click the Save button.

How harvesters work

1. A space is attached to the harvester (multiple spaces can be attached to one harvester).
2. The harvesting process collects metadata from all files in the space and places them in Elasticsearch indices assigned to the harvester.
3. Any metadata modifications, additions, deletions, or file movements — all these changes are dynamically updated in the indices.
4. The harvester provides a ready-to-use web application that allows querying the indices and presenting results to the user.
5. The Web app is pluggable; a custom implementation can be tailored to the needs of a specific user group.
6. The harvester can be made public. In that case, anyone — including non-Onedata users — can visit the metadata browser and query for the aggregated metadata.

Exercise

Attaching a space and harvesting metadata

1. Navigate to the harvester you have just created. Click on the Spaces tab in the sidebar.
2. The list of harvested spaces is empty. Click the Add one of your spaces button to attach one.
3. Choose the space alpha-11p from the dropdown and then click the Add button.
4. Go to the Indices view via the sidebar and click on the generic-index entry to expand it.
5. Wait until all progress indicators reach 100%.
6. Click on the Data discovery tab in the sidebar to bring up the metadata browser GUI.

Exercise

Querying harvested metadata

1. Locate the query builder at the top of the metadata browser. Use it to define the following query: __onedata.fileName.keyword is artem.jpg.

2. Click the Query button next to the query builder.

   The results will contain all entries matching your query. There should be multiple results showing files with the same name. The duplication is caused by every participant using the same set of files.

3. Try to build another query: imageCategory.keyword is city AND imageWidth > 1000. Click on the Query button to see the results.

   Play around with the constraints in the above query. You can adjust the values in the query builder — simply click on a value to modify. After applying changes, click outside of the editor to finalize.

Exercise

Updating harvested metadata

1. Go to the file browser of the space alpha-11p and enter your directory.

2. Open the metadata information for the file baskin.jpg (in FirstDataset/Photos), set a new basic metadata entry: favourite = true, and save changes.

3. Go back to the metadata browser GUI of your harvester.

4. Build a query, which will find a file you've just modified (use __onedata.xattrs.favourite.__value.keyword property) and click on the Query button.

   Your file should be present in the results list. If it isn't, try again after a few seconds — the harvesting process aggregates changes with a slight delay. Take a look at the indices tab to see the current aggregation status.