Data mappingsΒΆ

Once a topic hierarchy is defined, it needs to be included in the wis2box data mappings configuration. wis2box provides a default data mapping (in YAML format):

data:
    mwi.mwi_met_centre.data.core.weather.surface-based-observations.SYNOP:
        plugins:
            csv:
                - plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
                  template: synop-bufr.json
                  notify: true
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.csv$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
    ita.roma_met_centre.data.core.weather.surface-based-observations.SYNOP:
        plugins:
            bin:
                - plugin: wis2box.data.bufr4.ObservationDataBUFR
                  notify: true
                  file-pattern: '^.*\.bin$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
    dza.alger_met_centre.data.core.weather.surface-based-observations.SYNOP:
        plugins:
            bufr4:
                - plugin: wis2box.data.bufr4.ObservationDataBUFR
                  notify: true
                  buckets:
                    - ${WIS2BOX_STORAGE_INCOMING}
                  file-pattern: '^.*\.bufr4$'
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  buckets:
                    - ${WIS2BOX_STORAGE_PUBLIC}
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'

The data mappings are indicated by the data keyword, with each topic having a separate entry specifying:

  • plugins: all plugin objects associated with the topic, by file type/extension

Each plugin is based on the file extension to be detected and processed, with the following configuration:

  • plugin: the codepath of the plugin

  • notify: whether the plugin should publish a data notification

  • template: additional argument allowing a mapping template name to be passed to the plugin

  • file-pattern: additional argument allowing a file pattern to be passed to the plugin

  • buckets: the name(s) of the storage bucket(s) that data should be saved to (See Configuration for more informaiton on buckets)

The default data mapping can be overriden by user-defined data mappings with the following steps:

  • create a YAML file similar to the above to include your topic hierarchy

  • set the WIS2BOX_DATADIR_DATA_MAPPINGS environment variable to point to the new file of definitions

  • restart wis2box

See Extending wis2box for more information on adding your own data processing pipeline.