Too Long Didn’t Read¶
For the truly impatient, this section summarizes the steps to required to run wis2box from example configurations.
Environment¶
wis2box passes environment variables from dev.env to its containers on startup.
An example file is provided in examples/config/wis2box.env
.
Copy this file to your working directory, and update it to suit your needs.
cp examples/config/wis2box.env dev.env
Note
You must map WIS2BOX_HOST_DATADIR
to the absolute path of directory on your host machine.
For terminals that support environment variables, it may be useful to also define environment variables locally. For terminals that do not support environment variables, you will need to define the asolute path to your data directory.
# source
. dev.env
# verify
echo $WIS2BOX_HOST_DATADIR
Data mappings¶
To start processing your own data, you need to define a data mappings file. Example mapping files are provided in examples/config:
examples/config/synop-bufr-mappings.yml
, input is .bufr containing SYNOP observation-dataexamples/config/synop-csv-mappings.yml
, input is .csv containing SYNOP observation-data
To publish .bufr files of SYNOP data:
cp examples/config/synop-bufr-mappings.yml $WIS2BOX_HOST_DATADIR/data-mappings.yml
Edit $WIS2BOX_HOST_DATADIR/data-mappings.yml
and change [ISO3C_country].[center_id].data.core.weather.surface-based-observations.SYNOP
:
replace
ISO3C_country
with your corresponding ISO 3166 alpha-3 code.replace
center_id
with the string identifying the center running the wis2node.
Station metadata¶
wis2box needs to have a station_list.csv that contains the stations that will be sending data.
An example is provided in examples/config/station_list.csv
.
Update the file with your stations.
cp examples/config/station_list.csv $WIS2BOX_HOST_DATADIR/station_list.csv
Discovery metadata¶
Discovery metadata is central to wis2box, wis2box-api, and wis2box-ui.
An example is provided in examples/config/surface-weather-observations.yml
.
cp examples/config/surface-weather-observations.yml $WIS2BOX_HOST_DATADIR/surface-weather-observations.yml
Edit the file $WIS2BOX_HOST_DATADIR/surface-weather-observations.yml
to provide the correct metadata for your dataset:
replace
[ISO3C_country].[center_id].data.core.weather.surface-based-observations.SYNOP
with the topic you used in$WIS2BOX_HOST_DATADIR/data-mappings.yml
previouslytext provided in title and abstract will be displayed in wis2box-ui
provide a valid bounding-box in bbox
Build wis2box¶
Run the build
and update
commands to set up wis2box.
This will start the process of building the wis2box containers from source.
python3 wis2box-ctl.py build
python3 wis2box-ctl.py update
This might take a while the first time.
Start wis2box¶
Start wis2box containers and check that all services are running (and healthy).
python3 wis2box-ctl.py start
python3 wis2box-ctl.py status
If neccessary repeat the command until all services are up and running.
Runtime configuration¶
The last design-time steps required to run wis2box are once wis2box is running.
Login to the wis2box container
python3 wis2box-ctl.py login
Note
$WIS2BOX_DATADIR is the location that $WIS2BOX_HOST_DATADIR binds to the container. This allows wis2box command to access the configuration files from inside the wis2box container.
Setup observation data processing and API publication:
wis2box data add-collection $WIS2BOX_DATADIR/surface-weather-observations.yml
Cache and publish station collection and discovery metadata to the API:
wis2box metadata discovery publish $WIS2BOX_DATADIR/surface-weather-observations.yml
wis2box metadata station sync $WIS2BOX_DATADIR/station_list.csv
Logout of wis2box container:
exit
Data ingest¶
The runtime component of wis2box is data ingestion. This is an event driven workflow driven by s3 notifications from uploading data to wis2box-storage. An example is provided in examples/scripts/copy_to_incoming.py. To access the storage component, visit http://localhost:9001 in your web browser. The default username/password is minio/minio123
Debugging¶
Something’s now working? wis2box includes a local grafana-instance to help you collect and view logs and figure out what’s wrong. Visit http://localhost:3000 in your local web browser to view the local grafana instance.
wis2box-ui¶
wis2box includes a UI to view the data that has been ingested. To explore, visit http://localhost:8999 in your web browser.
Not seeing data? After data has been ingested for a station for the first time, you need to re-publish the stations. This will republish the station with a link relation to any associated observation collection.
python3 wis2box-ctl.py execute wis2box metadata station publish-collection