Storage¶

Overview¶

The default wis2box storage capability is powered by MinIO, which provides S3 compatible object storage.

The default wis2box MinIO administration user interface can be accessed locally at http://localhost:9001.

The username/password for MinIO is configured through environment variables (see Configuration).

Once logged in, buckets can be managed via the default “Buckets” menu item (click “Manage”). Click “Browse” provides a browsing capability for a storage administrator.

Uploading data to MinIO¶

Files can uploaded to the MinIO bucket in a number of ways. Any new file received on MinIO will trigger an MQTT notification which is received by wis2box.

Below are basic examples on sending data to the MinIO wis2box-incoming bucket. For more information and additional examples, consult the official MinIO documentation.

Using the boto3 Python Client¶

Install the Python boto3 package via pip:

pip3 install boto3

The below example copies a local file (myfile.csv) to the wis2box-incoming bucket with topic foo.bar.baz:

import boto3

endpoint_url = 'http://localhost:9000'
filename = 'myfile.csv'

session = boto3.Session(
    aws_access_key_id='wis2box',
    aws_secret_access_key='XXXXX' # your wis2box storage password
)

s3client = session.client('s3', endpoint_url=endpoint_url)

with open(filename, 'rb') as fh:
    s3client.upload_fileobj(fh, 'wis2box-incoming', f'foo/bar/baz/{filename}')

To allow uploading files into MinIO remotely, the wis2box-incoming bucket is proxied via Nginx.

For example, to upload the local file (WIGOS_0-454-2-AWSNAMITAMBO_2021-11-18T0955.csv with topic) to the folder mw-mw_met_centre/data/core/weather/surface-based-observations/synop:

import boto3

endpoint_url = '<your-wis2box-url>'
filename = 'WIGOS_0-454-2-AWSNAMITAMBO_2021-11-18T0955.csv'

session = boto3.Session(
    aws_access_key_id='wis2box',
    aws_secret_access_key='XXXXX' # your wis2box storage password
)

s3client = session.client('s3', endpoint_url=endpoint_url)

folder = 'mw-mw_met_centre/data/core/weather/surface-based-observations/synop'
with open(filename, 'rb') as fh:
    s3client.upload_fileobj(fh, 'wis2box-incoming', f'{folder}/{filename}')

Using the MinIO Python Client¶

MinIO provides a Python client which can be used as follows:

Install the Python minio module via pip:

pip3 install minio

The below example copies a local file (mydata.bin) to the wis2box-incoming bucket to topic foo.bar.baz:

from minio import Minio

filepath = '/home/wis2box-user/local-data/mydata.bin'
minio_path = '/it-roma_met_centre/data/core/weather/surface-based-observations/synop/'

endpoint = 'http://localhost:9000'
WIS2BOX_STORAGE_USERNAME = 'wis2box'
WIS2BOX_STORAGE_PASSWORD = 'XXXXX' # your wis2box storage password

client = Minio(
    endpoint=endpoint,
    access_key=WIS2BOX_STORAGE_USERNAME,
    secret_key=WIS2BOX_STORAGE_PASSWORD,
    secure=is_secure=False)

filename = filepath.split('/')[-1]
client.fput_object('wis2box-incoming', minio_path+filename, filepath)

Using S3cmd¶

Given MinIO is S3 compatible, data can be uploaded using generic S3 tooling. The below example uses S3cmd to upload data to wis2box MinIO storage:

Edit the following fields in ~/.s3cfg:

cat << EOF > ~/.s3cfg
# Setup endpoint
host_base = localhost:9000
use_https = False

# Setup access keys
access_key = wis2box
secret_key = XXXXX
EOF

Below is a simple command line example to copy a local file called myfile.csv into the wis2box-incoming bucket, to topic foo/bar/baz:

s3cmd myfile.csv s3://wis2box-incoming/foo/bar/baz

Using the MinIO UI¶

Files can also be uploaded interactively via the MinIO adminstration interface. The example below demonstrates this capability when browsing the wis2box-incoming bucket:

Uploading files using the MinIO adminstration interface

WIS2 in a box