Storage¶

Overview¶

The default wis2box storage capability is powered by MinIO, which provides S3 compatible object storage.

The default wis2box MinIO administration user interface can be accessed locally at http://localhost:9001.

The username/password for MinIO is configured through environment variables (see Configuration).

Once logged in, buckets can be managed via the default “Buckets” menu item (click “Manage”). Click “Browse” provides a browsing capability for a storage administrator.

Uploading data to MinIO¶

Files can uploaded to the MinIO bucket in a number of ways. Any new file received on MinIO will trigger an MQTT notification which is received by wis2box.

Below are basic examples on sending data to the MinIO wis2box-incoming bucket. For more information and additional examples, consult the official MinIO documentation.

Using the boto3 Python Client¶

Install the Python boto3 package via pip:

pip3 install boto3

The below example copies a local file (myfile.csv) to the wis2box-incoming bucket with topic foo.bar.baz:

import boto3

endpoint_url = '<your-wis2box-url>'
filename = 'myfile.csv'

session = boto3.Session(
    aws_access_key_id='wis2box',
    aws_secret_access_key='Wh00data!'
)

s3client = session.client('s3', endpoint_url=endpoint_url)

with open(filename, 'rb') as fh:
    s3client.upload_fileobj(fh, 'wis2box-incoming', f'foo/bar/baz/{filename}')

To allow uploading files into MinIO remotely, the wis2box-incoming bucket is proxied via Nginx.

For example, to upload the local file (WIGOS_0-454-2-AWSNAMITAMBO_2021-11-18T0955.csv with topic) with topic mwi.mwi_met_centre.data.core.weather.surface-based-observations.synop via the Nbinx proxy:

import boto3

endpoint_url = 'http://localhost:9000'
filename = 'myfile.csv'

session = boto3.Session(
    aws_access_key_id='wis2box',
    aws_secret_access_key='Wh00data!'
)

s3client = session.client('s3', endpoint_url=endpoint_url)

filename = 'WIGOS_0-454-2-AWSNAMITAMBO_2021-11-18T0955.csv'

with open(filename, 'rb') as f:
    s3client.upload_fileobj(f, 'wis2box-incoming', f'data/core/observations-surface-land/mw/FWCL/landFixed/{filename}')

Using the MinIO Python Client¶

MinIO provides a Python client which can be used as follows:

Install the Python minio module via pip:

pip3 install minio

The below example copies a local file (myfile.csv) to the wis2box-incoming bucket to topic foo.bar.baz:

from minio import Minio

client = Minio(
    'localhost:9000',
    access_key='minio',
    secret_key='minio123',
    secure=False
)

client.fput_object('wis2box-incoming', 'myfile.csv', '/foo/bar/baz/myfile.csv')

Using S3cmd¶

Given MinIO is S3 compatible, data can be uploaded using generic S3 tooling. The below example uses S3cmd to upload data to wis2box MinIO storage:

Edit the following fields in ~/.s3cfg:

cat << EOF > ~/.s3cfg
# Setup endpoint
host_base = localhost:9000
use_https = False

# Setup access keys
access_key = minio
secret_key = minio123
EOF

Below is a simple command line example to copy a local file called myfile.csv into the wis2box-incoming bucket, to topic foo/bar/baz:

s3cmd myfile.csv s3://wis2box-incoming/foo/bar/baz

Using the MinIO UI¶

Files can also be uploaded interactively via the MinIO adminstration interface. The example below demonstrates this capability when browsing the wis2box-incoming bucket:

Uploading files using the MinIO adminstration interface

WIS2 in a box