Quick-start - Containers


To get started with StorageFabric containers, use the following steps:

Installation and setup

Using StorageFabric


Create Buckets and Access Keys with Storage Providers

1 Create a configuration bucket with a storage provider

Amazon

Follow the Amazon S3 tutorial to create your Configuration Bucket in your desired AWS region. Note down the bucket name. We will use it later to setup StorageFabric.

Google

Follow the Google Cloud Storage tutorial to create your Configuration Bucket. Note down the bucket name. We will use it later to setup StorageFabric.

Azure

Follow the Microsoft Azure tutorial to create your Azure Cloud Storage Account.

Follow the Microsoft Azure tutorial to create an Azure container. This will be your Configuration Bucket. For Public access level, select the default level Private (no anonymous access). Note down the container name. We will use it later to setup StorageFabric.


2 Get your storage provider access credentials.

Create or obtain credentials for your storage providers so that StorageFabric can communicate with them.

Amazon

Follow the Amazon Security Credentials tutorial to create an Access Key ID and Secret Access Key with read-write access to your Configuration Bucket.

Google

Follow the Google Key Management tutorial to create an Access Key ID and Secret Access Key with read-write access to your Configuration Bucket.

Azure

Do the following to get an Access Key ID and a corresponding Secret Access Key for Azure.

Your storage account name will serve as an Access Key ID. Follow the Microsoft Azure tutorial to get keys for your Azure containers. The Storage account name will serve as the Access Key ID The key value (from either key1 or key2) will serve as the Secret Access Key



Setup StorageFabric

For scalability and ease-of-management, StorageFabric components are designed to be stateless. To remain stateless, along with user data, StorageFabric configuration data is also stored with storage providers (on-prem or in-cloud). The configuration data is encrypted using a Master Encryption Key. StorageFabric uses a designated Configuration Bucket to store its configuration data. We will setup StorageFabric with a designated Configuration Bucket here.

This model means that all you have to safeguard is your Master Encryption Key. All other keys and configuration are bootstrapped from the master key making key management significantly simpler.

1 Extract the StorageFabric archive

Extract the StorageFabric archive and change dir.

tar -xzvf storagefabric-3.3.0.tar.gz

cd storagefabric-3.3.0

Expected unarchived dir structure.

.
├── bin
├── config
│   ├── config.yaml
│   ├── license.yaml
│   ...
├── licenses
├── README.FIRST.txt
├── storagefabric-configuration-manager.tar.gz
├── storagefabric-gateway.tar.gz
└── storagefabric.tar.gz

2 Import StorageFabric containers

Import StorageFabric containers into your container platform.

Docker

docker load < storagefabric.tar.gz

Docker Hub

docker login

docker pull privatemachines/storagefabric:latest

Kubernetes - Docker Hub

# If you have access to the privatemachines Docker Hub registry
# all you need to do is generate a Kubernetes secret with your
# Docker Hub credentials. The provided registry_authentication.sh script
# will do this for you.
./bin/registry_authentication.sh '<DOCKER_USERNAME>' '<DOCKER_PASSWORD>'

Kubernetes - Local Docker Images

# This example shows how images can be imported into minikube.
# Steps may vary depending on your personal Kubernetes deployment
minikube ssh docker load < storagefabric-configuration-manager.tar.gz
minikube ssh docker load < storagefabric-gateway.tar.gz

3 Save your StorageFabric license

Docker

Save your StorageFabric license in the licenses/ folder.

Kubernetes

In config/license.yaml on the line immediately below license: |, paste the entirety of your license

Note

The license must be indented 2 spaces more than license: |

apiVersion: v1
kind: Secret
metadata:
  name: storagefabric-license
type: Opaque
stringData:
  license: |
    -----BEGIN CERTIFICATE-----
    THISISYOURLICENSETHISISYOURLICENSETHISISYOURLICENSETHISISYOURLIC
    THISISYOURLICENSETHISISYOURLICENSETHISISYOURLICENSETHISISYOURLIC
    THISISYOURLICENSETHISISYOURLICENSETHISISYOURLICENSETHISISYOURLIC
    .
    .
    .
    THISISYOURLICENSETHISISYOURLICENSETHISISYOURLICENSETHISISYOURLIC
    THISISYOURLICENSETHISISYOURLICENSETHISISYOURLICENSETHISISYOURLIC
    THISISYOURLICENSETHISISYOURLICENSETHISISYOURLICENSETHISISYOURLIC
    -----END CERTIFICATE-----

4 Generate a Master Encryption Key

Use the following command to create a Master Encryption Key:

./bin/generate_mek.sh

Warning

Save your Master Encryption Key. If not using a Key Management System (KMS), StorageFabric manages all your encryption and integrity keys. However, safeguarding and persisting the Master Encryption Key is your responsibility. To learn more about key management, see the full product documentation.


5 Add your configuration bucket settings

Within the storagefabric-3.3.0 directory, open the file config/config.yaml

Fill out all the uncommented lines with your data

Amazon

apiVersion: v1
kind: Secret
metadata:
    name: storagefabric-secret
type: Opaque
stringData:
    ACCESS_KEY_ID: # your AWS cloud access key ID
    SECRET_ACCESS_KEY: # your AWS secret access key
    STORAGEFABRIC_MASTER_ENCRYPTION_KEY: # your master encryption key (generated in the previous step)
    SESSION_TOKEN: "" # (optional) your AWS session token

---

apiVersion: v1
kind: ConfigMap
metadata:
    name: storagefabric-config
data:
    ADDRESSING_STYLE: virtual
    API_TYPE: s3_v4
    CONFIGURATION_BUCKET: # your AWS configuration bucket
    PROVIDER_URL:  # s3.amazonaws.com or region-specific url such as s3-eu-west-1.amazonaws.com
    PROVIDER_REGION: # your AWS s3 region. example: us-east-1 or eu-west-1

Google

apiVersion: v1
kind: Secret
metadata:
    name: storagefabric-secret
type: Opaque
stringData:
    ACCESS_KEY_ID: # your Google cloud access key ID
    SECRET_ACCESS_KEY: # your Google secret access key
    STORAGEFABRIC_MASTER_ENCRYPTION_KEY: # your master encryption key (generated in the previous step)
    SESSION_TOKEN: "" # (optional) your Google session token

---

apiVersion: v1
kind: ConfigMap
metadata:
    name: storagefabric-config
data:
    ADDRESSING_STYLE: path
    API_TYPE: s3
    CONFIGURATION_BUCKET: # your Google configuration bucket
    PROVIDER_URL:  storage.googleapis.com
    PROVIDER_REGION:

Azure

apiVersion: v1
kind: Secret
metadata:
    name: storagefabric-secret
type: Opaque
stringData:
    ACCESS_KEY_ID: # your Azure storage account
    SECRET_ACCESS_KEY: # your Azure access key
    STORAGEFABRIC_MASTER_ENCRYPTION_KEY: # your master encryption key (generated in the previous step)
    SESSION_TOKEN: "" # (optional) your Azure access token

---

apiVersion: v1
kind: ConfigMap
metadata:
    name: storagefabric-config
data:
    ADDRESSING_STYLE: path
    API_TYPE: azure
    CONFIGURATION_BUCKET: # your Azure container
    PROVIDER_URL:  # <Azure storage account>.blob.core.windows.net
    PROVIDER_REGION:

Run StorageFabric Containers

Docker

docker run \
  -p 30011:5600 -p 30022:8000 \
  -v $(pwd)/config/config.yaml:/etc/storagefabric/config.yaml \
  -v $(pwd)/licenses:/etc/storagefabric/licenses \
  --name storagefabric \
  privatemachines/storagefabric:latest

Note

To run the container the background, add the -d flag. Output from the container to STDOUT and STDERR can be retrieved with docker logs storagefabric

Note

To automatically remove the container when it exits, add the --rm flag.

Kubernetes

kubectl apply -f ./config

Wait a few moments for the StorageFabric to start up before connecting to the StorageFabric Configuration Manager or StorageFabric Gateway.


Configure StorageFabric

1. Connect to Configuration Manager Web UI

Note

By default, StorageFabric containers use self-signed certificates. Your web browser may flag this.

Kubernetes

http://nodeIP:30011/

2. Accept the End User License Agreement (EULA)

If you are connecting for the first time, you will be asked to read and accept the EULA. Scroll down and click the button I Accept to accept and continue.

3. Login to the Configuration Manager web UI

Default login
    username: admin
    password: password

4. Add cloud credentials for your Virtual Bucket to StorageFabric configuration.

Amazon

  • From the left-navigation bar, click on the link Cloud Access Keys.

  • Click the button Add Cloud Access Key.

  • In the Access Key ID field, enter your <AWS_ACCESS_KEY_ID>.

  • In the Secret Access Key field, enter your <AWS_SECRET_ACCESS_KEY>.

  • If you are using temporary Cloud Access Keys, also enter your Session Token in the Session Token field.

  • Leave the field Lifetime in minutes blank.

  • Check the option Admin Key. Checking this option means that StorageFabric can also use these credentials to create buckets with Amazon.

  • Click the button Add Cloud Access Key.

Google

  • From the left-navigation bar, click on the link Cloud Access Keys.

  • Click the button Add Cloud Access Key.

  • In the Access Key ID field, enter your <GOOGLE_ACCESS_KEY_ID>.

  • In the Secret Access Key field, enter your <GOOGLE_SECRET_ACCESS_KEY>.

  • Leave the field Session Token blank.

  • Leave the field Lifetime in minutes blank.

  • Check the option Admin Key. Checking this option means that StorageFabric can also use these credentials to create buckets with Google.

  • Click the button Add Cloud Access Key.

Azure

  • From the left-navigation bar, click on the link Cloud Access Keys.

  • Click the button Add Cloud Access Key.

  • In the Access Key ID field, enter your <AZURE_STORAGE_ACCOUNT>.

  • In the Secret Access Key field, enter your <AZURE_ACCESS_KEY>.

  • Leave the field Session Token blank.

  • Leave the field Lifetime in minutes blank.

  • Check the option Admin Key. Checking this option means that StorageFabric can also use these credentials to create containers with Azure.

  • Click the button Add Cloud Access Key.

5. Create a Virtual Bucket in the Virtual Buckets tab.

Amazon

  • From the left-navigation bar, click on the link Virtual Buckets.

  • Cick the button Create Virtual Bucket to open the Create Bucket form.

  • In the Virtual Bucket Name field, enter virtual-data-bucket. This is the Virtual Bucket Name.

  • In the Provider Name field, select amazon.

  • In the Cloud Bucket Name field, enter the name of your AWS Cloud Bucket. Note that this should be different than your Configuration Bucket. To automatically create the bucket with AWS, check the option Create Cloud Bucket. If bucket is already created at AWS, uncheck the option Create Cloud Bucket.

  • Select the Cloud Credentials tab.

  • In the Cloud Access Key ID field, select your <AWS_ACCESS_KEY_ID>.

  • Click the button Create.

Google

  • From the left-navigation bar, click on the link Virtual Buckets.

  • Cick the button Create Virtual Bucket to open the Create Bucket form.

  • In the Virtual Bucket Name field, enter virtual-data-bucket. This is the Virtual Bucket Name.

  • In the Provider Name field, select google.

  • In the Cloud Bucket Name field, enter the name of your Google Cloud Bucket. Note that this should be different than your Configuration Bucket. To automatically create the bucket with Google, check the option Create Cloud Bucket. If bucket is already created at Google, uncheck the option Create Cloud Bucket.

  • Select the Cloud Credentials tab.

  • In the Cloud Access Key ID field, select your <GOOGLE_ACCESS_KEY_ID>.

  • Click the button Create.

Azure

  • From the left-navigation bar, click on the link Providers

  • Cick the button Add Provider to open the Add Provider form.

  • In the Provider Name field, enter your <AZURE_STORAGE_ACCOUNT>

  • In the Provider Base URL field, enter your <AZURE_STORAGE_ACCOUNT>.blob.core.windows.net

  • In the Country field, enter US.

  • Leave the remaining fields in the Connections tab as default.

  • Select the API Settings tab.

  • In the API Type field, select AZURE.

  • Uncheck the Tail Range Supported checkbox.

  • In the Multipart Mode field, select Disabled.

  • Click the button Add.


  • From the left-navigation bar, click on the link Virtual Buckets.

  • Cick the button Create Virtual Bucket to open the Create Bucket form.

  • In the Virtual Bucket Name field, enter virtual-data-bucket. This is the Virtual Bucket Name.

  • In the Provider Name field, select your <AZURE_STORAGE_ACCOUNT>.

  • In the Cloud Bucket Name field, enter your Azure container name. Note that this should be different than your Configuration Bucket. To automatically create the container with Azure, check the option Create Cloud Bucket. If container is already created at Azure, uncheck the option Create Cloud Bucket.

  • Select the Cloud Credentials tab.

  • In the Cloud Access Key ID field, select your <AZURE_STORAGE_ACCOUNT>.

  • Click the button Create.

6. Create client credentials.

Access keys used by clients to authenticate to the StorageFabric Gateway are referred to as Client Access Keys.

  • From the left-navigation bar, expand the Client Access Keys section and then click on the link Client Access Keys.

  • Click the button Create Client Access Key to open the Create Client Access key form.

  • Click the button Create.

A new Access Key ID and Secret Access Key will be displayed. Copy and save them. We can use them to upload/download data to/from our Virtual Buckets.


Use the StorageFabric Gateway

Clients access the Virtual Bucket namespace across all storage providers via StorageFabric Gateways.

To access data in a unified manner, clients can use the StorageFabric File Browser, standard S3 api SDKs such as Java, Python (boto), C#, etc, command line tools such as s3cmd, or off-the-shelf file browsers such as S3 Browser.

Option 1: Use the StorageFabric File Browser

Access data directly from your web browser (Firefox, Chrome, etc.).

Kubernetes

http://<nodeIP>:30022/browser

On the login page, enter the ClientAccessKeyID and ClientSecretAccessKey generated in the previous section. You can now use the StorageFabric File Browser to securely upload and download data.

Option 2: Use s3cmd command line tool

In the following commands, replace <CLIENT_ACCESS_KEY_ID> and <CLIENT_SECRET_ACCESS_KEY> with he ClientAccessKeyID and ClientSecretAccessKey generated in the previous section.

Docker

# Create a sample data file
touch mydata.txt

# Upload to virtual bucket
s3cmd put mydata.txt s3://virtual-data-bucket/ \
    --host localhost:30022 \
    --host-bucket localhost:30022 \
    --access_key <CLIENT_ACCESS_KEY_ID> \
    --secret_key <CLIENT_SECRET_ACCESS_KEY> \
    --no-ssl

Kubernetes

# Create a sample data file
touch mydata.txt

# Upload to virtual bucket
s3cmd put mydata.txt s3://virtual-data-bucket/ \
    --host <nodeIP>:30022 \
    --host-bucket <nodeIP>:30022 \
    --access_key <CLIENT_ACCESS_KEY_ID> \
    --secret_key <CLIENT_SECRET_ACCESS_KEY> \
    --no-ssl

Option 3: Use SDKs

For instructions on using client-side sdks, refer to the full product documentation.


Shutdown StorageFabric

Docker

Stop the running container

docker stop storagefabric

Note

The docker run command used to start the StorageFabric container had the --rm flag, which means that the container will be removed once it is stopped.