Data mappingsΒΆ
Once a topic hierarchy is defined, it needs to be included in the wis2box data mappings configuration. wis2box provides a default data mapping (in YAML format):
data:
rou.rnimh.data.core.weather.surface-based-observations.synop:
plugins:
txt:
- plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
notify: true
file-pattern: '^A_SMR.*EDZW_(\d{4})(\d{2}).*.txt$'
bufr4:
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
file-pattern: '^A_SMR.*EDZW_(\d{4})(\d{2}).*.bufr4$'
mwi.mwi_met_centre.data.core.weather.surface-based-observations.synop:
plugins:
csv:
- plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
template: /data/wis2box/synop_bufr.json
notify: true
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.csv$'
bufr4:
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
ita.roma_met_centre.data.core.weather.surface-based-observations.synop:
plugins:
bin:
- plugin: wis2box.data.bufr4.ObservationDataBUFR
notify: true
file-pattern: '^.*\.bin$'
bufr4:
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
dza.alger_met_centre.data.core.weather.surface-based-observations.synop:
plugins:
bufr4:
- plugin: wis2box.data.bufr4.ObservationDataBUFR
notify: true
buckets:
- ${WIS2BOX_STORAGE_INCOMING}
file-pattern: '^.*\.bufr4$'
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
buckets:
- ${WIS2BOX_STORAGE_PUBLIC}
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
The data mappings are indicated by the data
keyword, with each topic having a separate entry specifying:
plugins
: all plugin objects associated with the topic, by file type/extension
Each plugin is based on the file extension to be detected and processed, with the following configuration:
plugin
: the codepath of the pluginnotify
: whether the plugin should publish a data notificationtemplate
: additional argument allowing a mapping template name to be passed to the plugin. Note that if the path is relative, the plugin must be able to locate the template accordinglyfile-pattern
: additional argument allowing a file pattern to be passed to the pluginbuckets
: the name(s) of the storage bucket(s) that data should be saved to (See Configuration for more information on buckets)
The default data mapping can be overriden by user-defined data mappings with the following steps:
create a YAML file similar to the above to include your topic hierarchy
place the file in the
WIS2BOX_DATADIR
directoryrestart wis2box
See Extending wis2box for more information on adding your own data processing pipeline.