Data mappingsΒΆ

Once a topic hierarchy is defined, it needs to be included in the wis2box data mappings configuration. wis2box provides a default data mapping (in YAML format):

data:
    cog.brazza_met_centre.data.core.weather.surface-based-observations.synop:
        plugins:
            txt:
                - plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
                  notify: true
                  file-pattern: '^.*_(\d{4})(\d{2}).*\.txt$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  buckets:
                    - ${WIS2BOX_STORAGE_PUBLIC}
                  file-pattern: '^.*\.bufr4$' 
    rou.rnimh.data.core.weather.surface-based-observations.synop:
        plugins:
            txt:
                - plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
                  notify: true
                  file-pattern: '^A_SMR.*EDZW_(\d{4})(\d{2}).*.txt$'
            csv:
                - plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
                  template: aws-template
                  notify: true
                  file-pattern: '^.*\.csv$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  file-pattern: '^A_SMR.*EDZW_(\d{4})(\d{2}).*.bufr4$'
    mwi.mwi_met_centre.data.core.weather.surface-based-observations.synop:
        plugins:
            csv:
                - plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
                  template: synop_bufr
                  notify: true
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.csv$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
    ita.roma_met_centre.data.core.weather.surface-based-observations.synop:
        plugins:
            bin:
                - plugin: wis2box.data.bufr4.ObservationDataBUFR
                  notify: true
                  file-pattern: '^.*\.bin$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
    dza.alger_met_centre.data.core.weather.surface-based-observations.synop:
        plugins:
            bufr4:
                - plugin: wis2box.data.bufr4.ObservationDataBUFR
                  notify: true
                  buckets:
                    - ${WIS2BOX_STORAGE_INCOMING}
                  file-pattern: '^.*\.bufr4$'
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  buckets:
                    - ${WIS2BOX_STORAGE_PUBLIC}
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'

The data mappings are indicated by the data keyword, with each topic having a separate entry specifying:

  • plugins: all plugin objects associated with the topic, by file type/extension

Each plugin is based on the file extension to be detected and processed, with the following configuration:

  • plugin: the codepath of the plugin

  • notify: whether the plugin should publish a data notification

  • template: additional argument allowing a mapping template name to be passed to the plugin. Note that if the path is relative, the plugin must be able to locate the template accordingly

  • file-pattern: additional argument allowing a file pattern to be passed to the plugin

  • buckets: the name(s) of the storage bucket(s) that data should be saved to (See Configuration for more information on buckets)

The default data mapping can be overriden by user-defined data mappings with the following steps:

  • create a YAML file similar to the above to include your topic hierarchy

  • place the file in the WIS2BOX_DATADIR directory

  • restart wis2box

See Extending wis2box for more information on adding your own data processing pipeline.