Downloading data from WIS2 using pywis-pubsub

Overview

This section provides examples of using the pywis-pubsub tool to download data from WIS2 global services. WIS2 global services include a Global Broker that provides users the ability to subscribe to data (via topics) and download to their local environment / workstation / decision support system from the WIS2 Global Cache.

The pywis-pubsub tool

This repository includes docker files to help you subscribe and download data from the WIS2 network, by using pywis-pubsub inside the wis2box-subscribe-download container.

pywis-pubsub is a Python package that provides publish, subscription and download capability of data from WIS2 infrastructure services.

Before starting the wis2box-subscribe-download container, the default configuration (provided in docker/wis2box-subscribe-download/local.yml) must be updated, by defining the URL of the MQTT broker as well as the desired topics to subscribe to.

In addition, the storage path should be updated to specify where the downloaded should be saved to.

# fully qualified URL of broker
broker: mqtts://username:password@host:port

# whether to run checksum verification when downloading data (default true)
verify_data: true

# whether to validate broker messages (default true)
validate_message: true

# list of 1..n topics to subscribe to
topics:
    - 'cache/a/wis2/topic1/#'
    - 'cache/a/wis2/topic2/#'

# storage: filesystem
storage:
    type: fs
    options:
        path: /tmp/foo/bar

To start a continuous subscribe-and-download process, run the wis2box-subscribe-download container as follows (-d for detached mode, --build to ensure changes in local.yml are built into the container):

docker-compose -f docker/docker.subscribe-download.yml up -d --build

To stop the subscribe-and-download process, run the following command:

docker-compose -f docker/docker.subscribe-download.yml down

Running pywis-pubsub interactively

pywis-pubsub can also be run interactively from inside the wis2box main container as follows:

# login to wis2box main container
python3 wis2box-ctl.py login

# edit a local configuration by using docker/wis2box-subscribe-download/local.yml as a template
vi /data/wis2box/local.yml

# connect, and simply display data notifications
pywis-pubsub subscribe --config local.yml

# connect, and additionally download messages
pywis-pubsub subscribe --config local.yml --download

# connect, and filter messages by bounding box geometry
pywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84

Summary

The above examples provide examples of using pywis-pubsub to subscribe and download data from WIS2 global services.