The installation via Docker Compose follows the approach – one Semantic Turkey and one GraphDB dedicated, respectively, to ShowVoc and VocBench. This means that the two systems have separate projects. On the contrary, in the deployment we previously had (made and managed by Andrea), VocBench and ShowVoc use the same Semantic Turkey and the same GraphDB, with the consequence that to publish a project it was enough to add the user showvoc-public (but on the project dashboard one still need to create/update the various indexes).

Overview

The picture below summaries the current data flow (read the graphic by starting from VocBench, in the middle). 

Getting data in VocBench.
So far, this is done by CC via GUI.

Bulk load. Not implemented, but useful in case of recovery or to populate a new instance (e.g., FAO prod, instances at BC3). TO be DONE. 

Folder with all files (all classifications, all formats), used by CC and CM: https://console.cloud.google.com/storage/browser/fao-datalab-caliper;tab=objects?authuser=1&prefix=&forceOnObjectsSortingFiltering=false

Getting data in ShowVoc,
Now, CC does it manually, from GUI (it consists of "deployng" VB projects into SV – see section below in this page). Craig suggests that it could be done via script too. TODO later. 

From VB/GraphDB to Fuseki:

So far, done by CM via command line (=one project at a time). A script for automatic export from VB → import in Fuseki was passed on by Andrea UNITOV, adapted by CM (schedule, 12:00, 19:00). Data is now loaded in Fuseki, but in a somewhat problematic way, as it is now, the the queries developed so far need to be adjusted in order to keep working. CC contacted UNITOV for advise on how to fix this. Craig will implement. 

From VocBench to ShowVoc

To make a VB project visible in ShowVoc, that must be “deployed”. To do that,

1. Go to: "Global Data Management / Export data"
2. In Deployment, choose "Deploy to triple store", Deployer = "ShowVoc Deployer". The deployment can be configured either as "Deploy to new project" or "Deploy to existing project".
   
   For New project, instructions follow (screenshot below).
   
   
   
   
   
3. After selecting Deploy to new project, click Configure:
   1. add the URL of the Semantic Turkey associated to your ShowVoc: http://showvoc:1979/ (this is the hostname configured by Docker, it should be the same in the corresponding installation).
   2. username and password of a user with admin rights
   3. name of the new project to create
   4. Core repository configuration = "GraphDB 10+ (remote only)" (does not need to be configured)
   5. click OK
      
      
      
      
4. Click Submit
   
   
   
   
5. Now, to have the project visible in ShowVoc, you need to login to ShowVoc, as an administrator, and go to the "Administration dashboard" and make the project Public (now is Staging):
   
   
   
   
6. Make the project Public
   
   
   
   
7. After having selected Public, a window is prompted where you need to check the boxes to Create index and Create dataset metadata for the project.
   
   

Now, you can log out and go back to the home page of ShowVoc where you will find the new dataset.

Important. When you deploy to an existing project, you still need to go to the ShowVoc admin dashboard and update indexes and recompute the metadata.

How to simplify the process described above. [Instructions not tested yet]

In VocBench, it is possible to create a super user account and save its credential as default. In this way, when you submit a new dataset from VocBench to ShowVoc, you will no longer prompted to insert your credential. The same solution can be applied to the configuration of the triple store when deploying new projects. This can be done in the section Administration in VocBench.

Then, go configure the ShowVoc linked. See the steps below.

At step 3:

• select scope SYSTEM:

• API base URL: http://showvoc:1979/

• Frontend base URL: here goes the "external" address (given by the proxy) of the web app di Showvoc. For the Datalab, that is: https://datalab.review.fao.org/caliper/browse/showvoc/ For UNITOV:  http://terra.art.uniroma2.it:9090/browse/showvoc/
• username e password: credential mentioned above..

Click button Submit (not visible in the screenshot) to save the data just inserted.

Then, go to scope PROJECT and as "Core repository configuration" select " GraphDB 10+ (remote only)". Click submit again.

If you now redo all steps, you will find all fields populated, except for the one of the project. Note that the credentials are not shown, nor they are ever eturned to the client.

Unfortunately, the name of the project needs to be inserted every time. In order to avoid mistakes, it is advisable to use the same project name for VocBench and for ShowVoc.

From VocBench to Fuseki

Data can be moved from VocBench to Fuseki programmatically using the scripts in the scripts/data-flow folder of the https://bitbucket.org/cioapps/fao-datalab-caliper-docker repository. Once you have cloned the repository, you simply have to run the following command with the appropriate parameters to move data from VocBench to Fuseki.

The arguments for the vocbench_to_fuseki.py script are as follows:

• --vocbench_url (type=str), default="http://localhost:8080/caliper/edit/vocbench3/" - This is the VocBench3 url associated to your instance.
• --email (type=str) - This is the VocBench login email address.
• --password (type=str) - This is the VocBench login password.
• --project_names (type=str), default="all" - This is the comma separated names of the projects to export. Project name is case sensitive. Use "all" to export and import all project excluding those with "test" or "staging" in their name.
• --remove_rdfs (type=bool), default=False - Whether to remove the exported rdf files after import.
• --fuseki_bin (type=str), default="fuseki-bin" - This is the /bin folder for Fuseki ruby scripts.
• --fuseki_url (type=str), default="http://localhost:3030/ds" - help='This is the Fuseki url.

In most cases you can leave the parameters as default if running from the same machine as the Caliper instance.

# from the data-flow folder run the following command;  
python3 vocbench_to_fuseki.py --email '<VOCBENCH_EMAIL>' --password '<VOCBENCH_PASSWORD>' --project_names 'all' --remove_rdfs True

# for the VOCBENCH_PASSWORD especially, when running from a bash terminal, it's best to encase the password in single quotes as special characters such as @ and & in your password can cause it to fail;