Data mappingsΒΆ
Once a topic hierarchy is defined, it needs to be included in the wis2box data mappings configuration. wis2box provides a default data mapping (in YAML format):
data:
mwi.mwi_met_centre.data.core.weather.surface-based-observations.SYNOP:
plugins:
csv:
- plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
template: synop-bufr.json
notify: true
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.csv$'
bufr4:
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
ita.roma_met_centre.data.core.weather.surface-based-observations.SYNOP:
plugins:
bin:
- plugin: wis2box.data.bufr4.ObservationDataBUFR
notify: true
file-pattern: '^.*\.bin$'
bufr4:
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
dza.alger_met_centre.data.core.weather.surface-based-observations.SYNOP:
plugins:
bufr4:
- plugin: wis2box.data.bufr4.ObservationDataBUFR
notify: true
buckets:
- ${WIS2BOX_STORAGE_INCOMING}
file-pattern: '^.*\.bufr4$'
- plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
buckets:
- ${WIS2BOX_STORAGE_PUBLIC}
file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'
The data mappings are indicated by the data
keyword, with each topic having a separate entry specifying:
plugins
: all plugin objects associated with the topic, by file type/extension
Each plugin is based on the file extension to be detected and processed, with the following configuration:
plugin
: the codepath of the pluginnotify
: whether the plugin should publish a data notificationtemplate
: additional argument allowing a mapping template name to be passed to the pluginfile-pattern
: additional argument allowing a file pattern to be passed to the pluginbuckets
: the name(s) of the storage bucket(s) that data should be saved to (See Configuration for more informaiton on buckets)
The default data mapping can be overriden by user-defined data mappings with the following steps:
create a YAML file similar to the above to include your topic hierarchy
set the
WIS2BOX_DATADIR_DATA_MAPPINGS
environment variable to point to the new file of definitionsrestart wis2box
See Extending wis2box for more information on adding your own data processing pipeline.