Ingesting data for publication

In WIS2, the availability of new data to be downloaded is announced using WIS2 Notifications sent using the MQTT protocol. Each WIS2 Notification will contain a “canonical” URL to the data to enable the data to be downloaded over HTTP(S).

The wis2box-management service listens to updates from the wis2box-storage-service about new files received and will attempt to process the files based on the datasets and data mappings that have been configured in the previous section.

The wis2box-storage service is based on MinIO , the following buckets are pre-configured when the wis2box-stack is started for the first time:

• wis2box-incoming: this bucket is used to received incoming files and are used as input by the data mappings configured in your wis2box

• wis2box-public: this bucket is used to store files to be shared on WIS2, it is proxied by the web-proxy service to make it available over HTTP(S) at WIS2BOX_URL/data/

Note

If you use “CSV to BUFR” plugin in your data mappings, the columns defined in the input CSV file must match the columns defined by the csv2bufr-template. If the columns do not match, the data will not be processed and an error will be raised in the logs of the wis2box-management container.

See the wis2box.data.csv2bufr.ObservationDataCSV2BUFR for more information on this data plugin and see the csv2bufr-templates repository for the template definitions.

Note

If you use “FM-12 to BUFR” (synop2bufr) plugin in your data mappings, your filename must contain a year and month, and the year and month should match the 1st and 2nd group in your regular expression (regex).

See the wis2box.data.synop2bufr.SYNOP2BUFR for more information.

For production use, it is recommended to set up an automated workflow and to regularly review the data publication workflow in the Grafana-service.

Data ingest methods

• Manually using the webapp
• Using custom scripts
• Using MiniO SFTP
• Using Moving Weather
• Using SURFACE
• Using ClimSoft
• Using CLiDE
• Using ADL
• Using CR1000X over MQTT

Testing and monitoring

• Testing using MinIO Console
• Data ingest workflow monitoring

Next steps

After you have successfully set up your data ingest process into the wis2box, you are ready to share your data with the global WIS2 network by enabling external access to your public services.

Next: Public services setup