Extending wis2box

At its core, wis2box is a plugin architecture orchestrating all the required components of a node in the WIS2 network. Driven by topic hierarchies, wis2box can be used to process and publish any type of geospatial data beyond the requirements of the WIS2 itself.

In this section we will to explore how wis2box can be extended. wis2box plugin development requires knowledge of how to program in Python as well as Python’s packaging and module system.

Building your own data plugin

The heart of a wis2box data plugin is driven from the wis2box.data.base abstract base class (ABC) located in wis2box/data/base.py. Any wis2box plugin needs to inherit from wis2box.data.base.BaseAbstractData. A minimal example can be found below:

from datetime import datetime
from wis2box.data.base import BaseAbstractData

class MyCoolData(BaseAbstractData):
    """Observation data"""
    def __init__(self, defs: dict) -> None:
        super().__init__(defs)

    def transform(self, input_data: Path) -> bool:
        # transform data
        # populate self.output_data with a dict as per:
        self.output_data = {
            'c123': {
                '_meta': {
                    'identifier': 'c123',
                    'relative_filepath': '/path/to/item/',
                    'data_date': datetime_object
                },
                'bufr4': bytes(12356),
                'geojson': geojson_string
            }
        }
        return True

The key function that plugin needs to implement is the transform function. This function should return a True or False of the result of the processing, as well as populate the output_data property.

The output_data property is a dict of keys/values. Each key should be the identifier of the item, with the following values dict:

The _meta element can include the following:
  • identifier: identifier for report (WIGOS_<WSI>_<ISO8601>)

  • relative_filepath: path to data, required to publish data with BaseAbstractData.publish

  • geometry: GeoJSON geometry object, required to send geometry with WIS2 notification

  • md5: md5 checksum of encoded data

  • wigos_station_identifier: WIGOS identifier

  • data_date: (as Python datetime objects) based on the observed datetime

  • originating_centre: Originating centre (see Common code table C11)

  • data_category: Category of data, see BUFR Table A

  • <format-extension>: 1..n properties for each format representation, with the key being the filename extension. The value of this property can be a string or bytes, depending on whether the underlying data is ASCII or binary, for example

Packaging

The next step is assembling your plugin using standard Python packaging. All plugin code and configuration files should be made part of the package so that it can operate independently when running in wis2box. For distribution and installation, you have the following options:

  • publish to the Python Package Index (PyPI) and install in the wis2node container with pip3 install wis2box-mypackage

  • git clone or download your package, and install via python3 setup.py install

See the Python packaging tutorial or Cookiecutter PyPackage for guidance and templates/examples.

Note

It is recommended to name your wis2box packages with the convention wis2box-MYPLUGIN-NAME, as well as adding the keywords/topics wis2box and plugin to help discovery on platforms such as GitHub.

Integration

Once your package is installed on the wis2box-management container, the data mappings need to be updated to connect your plugin to a topic hierarchy. See Data mappings for more information.

An example plugin for proof of concept can be found in https://github.com/wmo-cop/wis2box-csv-observations

Example plugins

The following plugins provide useful examples of wis2box plugins implemented by downstream applications.

Plugin(s)

Organization/Project

Description

wis2box-csv-observations

WMO

plugin for CSV surface observation data

wis2box-pyopencdms-plugin

OpenCDMS

plugin for connecting the Open Climate Data Management System to wis2box