Installation and configuration

This section summarizes the steps required to install a wis2box instance and setup your own datasets using initial configuration files provided by using the wis2box-create-config.py script.

Ensure you have Docker, Docker Compose and Python installed on your host, as detailed in Getting started.

Download

Download the wis2box setup files from the wis2box Releases page. Go to the latest release and download the wis2box-setup-1.0b6.zip file from the Assets section.

wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b6.zip
unzip wis2box-setup-1.0b6.zip
cd wis2box-1.0b6

Create initial configuration files

Run the following command to create the initial configuration files for your wis2box:

python3 wis2box-create-config.py

Note

The wis2box-create-config.py script will ask for a directory to store the configuration files. Please provide the absolute path to the directory where you want to store the configuration files, for example /home/wis2box-user/wis2box-data. This directory will be mapped to /data/wis2box inside the wis2box-management container.

The script will also ask for the URL of your wis2box. Please provide the public URL of your wis2box, for example http://mywis2box.example.com. For testing purpose you can also provide the internal IP address you use to access the host, for example http://192.168.0.3 and you change the URL in configuration files at a later point in time.

The script will propose to automatically create passwords for WIS2BOX_STORAGE_PASSWORD and WIS2BOX_BROKER_PASSWORD. These passwords are for internal use only within the wis2box, and it is recommended to accept the randomly generated passwords.

The script will ask for your TLD, the top level domain of your country, for example fr for France. International organizations should use int.

It will also ask for a centre-id. Please provide a lowercase string delimited by -, that is comprised of the TLD and a lowercase string that identifies your organization and does not use spaces or special characters, for example fr-meteofrance.

Note

In alignment with WMO WIS2 Guidance, the centre-id is automatically converted to lowercase when saved in wis2box.

The remaining questions will be used in the creation the discovery metadata files for the synop and temp datasets.

Discovery metadata

Discovery metadata provides the data description needed for users to discover your data when searching the WIS2 Global Discovery Catalogue.

The discovery metadata is provided in the form of a YAML file.

If you used the python3 wis2box-create-config.py command to initialize your wis2box, you will find two initial the discovery metadata files in the directory you specified for your configuration files, under the metadata/discovery/ directory:

  • metadata-synop-centreid.yml: contains the discovery metadata for the synop dataset

  • metadata-temp-centreid.yml: contains the discovery metadata for the temp dataset

Where centreid is the centre-id you specified during the configuration step.

Please review the content of these files and edit them as needed.

You can also add additional discovery metadata files for any other datasets you wish to publish.

Starting wis2box

Once you have prepared the necessary configuration files as described above you are ready to start the wis2box.

Run the following command to start wis2box:

python3 wis2box-ctl.py start

This might take a while the first time, as Docker images will be downloaded.

Note

The wis2box-ctl.py program is used as a convenience utility around a set of Docker Compose commands. You can customize the ports exposed on your host by editing docker-compose.override.yml.

Note

If you get the error:

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Please ensure your username is added to the docker group sudo usermod -aG docker <your-username>. Logout and log back in so that your group membership is re-evaluated.

Once the command above is completed, check that all services are running (and healthy).

python3 wis2box-ctl.py status

Which should display the following:

         Name                       Command                  State                           Ports
-----------------------------------------------------------------------------------------------------------------------
cadvisor                 /usr/bin/cadvisor -logtostderr   Up (healthy)   8080/tcp
elasticsearch            /bin/tini -- /usr/local/bi ...   Up (healthy)   9200/tcp, 9300/tcp
grafana                  /run.sh                          Up             0.0.0.0:3000->3000/tcp
loki                     /usr/bin/loki -config.file ...   Up             3100/tcp
mosquitto                /docker-entrypoint.sh /usr ...   Up             0.0.0.0:1883->1883/tcp, 0.0.0.0:8884->8884/tcp
mqtt_metrics_collector   python3 -u mqtt_metrics_co ...   Up             8000/tcp, 0.0.0.0:8001->8001/tcp
nginx                    /docker-entrypoint.sh ngin ...   Up             0.0.0.0:80->80/tcp
prometheus               /bin/prometheus --config.f ...   Up             9090/tcp
wis2box                  /entrypoint.sh wis2box pub ...   Up
wis2box-api              /app/docker/es-entrypoint.sh     Up
wis2box-auth             /entrypoint.sh                   Up
wis2box-minio            /usr/bin/docker-entrypoint ...   Up (healthy)   0.0.0.0:9000->9000/tcp, 0.0.0.0:9001->9001/tcp
wis2box-ui               /docker-entrypoint.sh ngin ...   Up             0.0.0.0:9999->80/tcp

Refer to the Troubleshooting section if this is not the case.

You should now be able to view collections on the wis2box API by visiting the URL you specified during the configuration step,

and adding /oapi/collections to the URL.

Initial wis2box API collections list

The API will show one (initially empty) collection named ‘Data Notifications’.

This collection will be filled when you start ingesting data and publishing WIS2 notifications.

Note

Additional collections will be added during the runtime configuration.

Runtime configuration

The following last design time steps are then required once wis2box is running.

Login to the wis2box-management container

python3 wis2box-ctl.py login

The first step is to pubished your dataset using the discovery metadata file you created above.

For example to add the dataset metadata/discovery/metadata-synop-centreid.yml in the directory you specified for your configuration files:

wis2box dataset publish /data/wis2box/metadata/discovery/metadata-synop-centreid.yml

This command publishes an MQTT message with information about your dataset to the WIS2 Global Discovery Catalogue. Repeat this command whenever you have to provide updated metadata about your dataset.

You can review the discovery metadata just cached through the new item at /oapi/collections/discovery-metadata/items:

You can view the dataset you just added in the API, by re-visiting /oapi/collections in a web browser.

Repeat this step for any other discovery metadata you wish to publish, such as the temp dataset.

Note

To ensure that discovery metadata is shared with the WIS2 Global Discovery Catalogue, you must ensure that your discovery metadata is published once the WMO Global Broker is connected and subscribed for your wis2box broker. If you have published discovery metadata before this stage, you must re-publish using the command above.

wis2box API collections list with added collection

Finally it is recommended to prepare authentication tokens for updating your stations and ingesting data using the wis2box-webapp.

To create a token for ingesting data:

wis2box auth add-token --path processes/wis2box

Record the token value displayed in a safe place, you will need for the Data ingest setup.

To create a token for updating stations:

wis2box auth add-token --path collections/stations

Record the token value displayed in the output of the command above. You will use this token to update stations in the next section.

You can now logout of wis2box-management container:

exit

Adding station metadata

The next step is to add station metadata to your wis2box. This can be done interactively in the wis2box-webapp UI on the ‘stations’ page.

The wis2box-webapp can be accessed by visiting the URL you specified during the configuration step, and adding /wis2box-webapp to the URL.

wis2box webapp stations page

Please note only data for stations that have been added to the wis2box will be ingested and result in WIS2 notifications being published.

Bulk inserting stations from CSV

You can also bulk insert a set of stations from a CSV file, by defining the stations in metadata/stations/station_list.csv in your wis2box host directory and running the following command:

python3 wis2box-ctl.py login
wis2box metadata station publish-collection

After doing a bulk insert please review the stations in wis2box-webapp and associate each station to the correct topics.

Adding topics to stations from the command line

If you want to associate all stations in your station metadata to one topic, you can use the following command:

python3 wis2box-ctl.py login
wis2box metadata station add-topic <topic-id>

If you want to add a topic to a single station, you can use the following command:

python3 wis2box-ctl.py login
wis2box metadata station add-topic --wsi <station-id> <topic-id>

If you want to add a topic to all stations from a specific territory, for example Italy, you can use the following command:

python3 wis2box-ctl.py login
wis2box metadata station add-topic --territory-name Italy <topic-id>

The next is the Data ingest setup.