Quick-start - VMs and Barebone

Welcome to the StorageFabric quick-start guide. This guide provides instructions to getting started with StorageFabric using the StorageFabric software repository.

For more detailed instructions, refer to the full product documentation. Prior to using this guide, you may also be interested in learning the full product documentation.

To get started with StorageFabric, take the following steps:

Create Buckets and Access Keys with Storage Providers
Install StorageFabric
Setup StorageFabric
Configure StorageFabric
Upload files securely using StorageFabric

Create Buckets and Access Keys with Storage Providers

Create buckets where your StorageFabric configuration data and your user data will be stored. With StorageFabric, all data (including configuration) is stored with providers. This makes StorageFabric components stateless, resulting in less management overhead for users and enabling quick disaster recovery.

1 Create a configuration bucket and a data bucket with a storage provider.

Amazon

Follow the Amazon S3 tutorial to create your Configuration Bucket in your desired AWS region. Note down the bucket name. We will use it later to setup StorageFabric.

Google

Follow the Google Cloud Storage tutorial to create your Configuration Bucket. Note down the bucket name. We will use it later to setup StorageFabric.

Azure

Follow the Microsoft Azure Create a storage account tutorial to create your Azure Cloud Storage Account.

Follow this Microsoft Azure tutorial to create an Azure container. This will be your Configuration Bucket. For Public access level, select the default level Private (no anonymous access). Note down the container name. We will use it later to setup StorageFabric.

2 Get your storage provider access credentials.

Create or obtain credentials for your storage providers so that StorageFabric can communicate with them.

Amazon

Follow the Amazon Security Credentials tutorial to create an Access Key ID and Secret Access Key with read-write access to your Configuration Bucket.

Google

Follow the Google Key Management tutorial to create an Access Key ID and Secret Access Key with read-write access to your Configuration Bucket.

Azure

Take the following steps to obtain an Access Key ID and a corresponding Secret Access Key for Azure:

Your storage account name will serve as an Access Key ID.
Follow the Microsoft Azure tutorial to obtain keys for your Azure containers.
The storage account name will serve as the Access Key ID
The key value (from either key1 or key2) will serve as the Secret Access Key

Next Step: Install StorageFabric

Install StorageFabric

Using the Virtalica Repository

Note

Prior to proceeding with repository setup, you must create buckets and access keys as described in the Section Create Buckets and Access Keys with Storage Providers.

1 Get access to the Virtalica repository.

To access StorageFabric software packages, you must obtain access to the Virtalica repository. For a step-by-step guide to setting up repository access, refer to the tutorial the full product documentation.

2 Install the StorageFabric Components.

2a. Install the StorageFabric Configuration Manager.

sudo yum install storagefabric-configuration-manager

sudo storagefabric-configuration-manager-post-install

2b. Install the StorageFabric Gateway.

sudo yum install storagefabric-gateway

sudo storagefabric-gateway-post-install

Next Step: Set up StorageFabric.

Set up StorageFabric

1 Change to the storagefabric user.

sudo su storagefabric

2 Save license file.

In order to use StorageFabric, you must have a valid License. Prior to setting up the Configuration Manager, save your license file in the folder /etc/storagefabric/licenses/.

3 Set Up StorageFabric Components.

3a. Set up the StorageFabric Configuration Manager.

Set up the StorageFabric Configuration Manager with your Configuration Bucket and access keys.

First, generate a new Master Encryption Key with the following command:

storagefabric-keygen --master-encryption-key

A new Master Encryption Key will be displayed. For example:

D9B2B4F1C3C8C1392A99B8DD326A47F3095F42FB9E037BD9E44AF8AA972B65095AED2D23

Then, setup the Configuration Manager for your chosen provider:

Amazon

storagefabric-configuration-manager-setup \
    --master-encryption-key <MASTER_ENCRYPTION_KEY> \
    --configuration-provider-url s3.amazonaws.com \
    --configuration-bucket <AWS_CONFIGURATION_BUCKET> \
    --configuration-access-key-id <AWS_ACCESS_KEY_ID> \
    --configuration-secret-access-key <AWS_SECRET_ACCESS_KEY>

In the above command, replace

<MASTER_ENCRYPTION_KEY> with the Master Encryption Key from Step 2a.
<AWS_CONFIGURATION_BUCKET> with your AWS configuration bucket.
<AWS_ACCESS_KEY_ID> with your Amazon access key ID.
<AWS_SECRET_ACCESS_KEY> with your Amazon secret access key.

Google

storagefabric-configuration-manager-setup \
  --master-encryption-key <MASTER_ENCRYPTION_KEY> \
  --configuration-provider-url storage.googleapis.com \
  --configuration-bucket <GOOGLE_CONFIGURATION_BUCKET> \
  --configuration-access-key-id <GOOGLE_ACCESS_KEY_ID> \
  --configuration-secret-access-key <GOOGLE_SECRET_ACCESS_KEY>

In the above command, replace

<MASTER_ENCRYPTION_KEY> with the Master Encryption Key from Step 2a.
<GOOGLE_ACCESS_KEY_ID> with your Google access key ID.
<GOOGLE_SECRET_ACCESS_KEY> with your Google secret access key.

Azure

storagefabric-configuration-manager-setup \
  --master-encryption-key <MASTER_ENCRYPTION_KEY> \
  --configuration-provider-url <AZURE_STORAGE_ACCOUNT>.blob.core.windows.net \
  --configuration-bucket <AZURE_CONFIGURATION_BUCKET> \
  --configuration-access-key-id <AZURE_STORAGE_ACCOUNT> \
  --configuration-secret-access-key <AZURE_ACCESS_KEY> \
  --configuration-provider-api-type azure

In the above command, replace

<MASTER_ENCRYPTION_KEY> with the Master Encryption Key from Step 2a.
<AZURE_STORAGE_ACCOUNT> with your Azure storage account.
<AZURE_ACCESS_KEY> with your Azure container access key.

Note

If you want this setup to persist after your machine is rebooted, the --persist flag is required.

Note

If you make a mistake and want to run the storagefabric-configuration-manager-setup again, the --reset-local flag is required to overwrite existing configuration.

3b. Get the View Encryption Key.

Get the View Encryption Key. We will use it in the next step to setup the StorageFabric Gateway.

storagefabric-configuration-manager-setup -V --include-view-keys

The View Encryption Key for the default view will be displayed similar to the following

Encrypted View Encryption Key for view default: U2FsdGVkX18IR....S3P5/sM9ueS5L

Note down the displayed Encryption Key for the default view. We will use it in the next step to setup the Gateway.

3c. Set up the StorageFabric Gateway.

Set up the StorageFabric Gateway with your View Encryption Key, Configuration Bucket, and access keys.

Amazon

storagefabric-gateway-setup \
  --gateway-domain s3.abccorp.com \
  --view default \
  --encrypted-view-encryption-key U2FsdGVkX18IR....S3P5/sM9ueS5L \
  --configuration-provider-url s3.amazonaws.com \
  --configuration-bucket <AWS_CONFIGURATION_BUCKET> \
  --configuration-access-key-id <AWS_ACCESS_KEY_ID> \
  --configuration-secret-access-key <AWS_SECRET_ACCESS_KEY>

In the above command, replace

<MASTER_ENCRYPTION_KEY> with the Master Encryption Key from Step 2a.
<AWS_CONFIGURATION_BUCKET> with your configuration bucket name.
<AWS_ACCESS_KEY_ID> with your Amazon access key ID.
<AWS_SECRET_ACCESS_KEY> with your Amazon secret access key.

Google

storagefabric-gateway-setup \
  --gateway-domain s3.abccorp.com \
  --view default \
  --encrypted-view-encryption-key U2FsdGVkX18IR....S3P5/sM9ueS5L \
  --configuration-provider-url storage.googleapis.com \
  --configuration-bucket <GOOGLE_CONFIGURATION_BUCKET> \
  --configuration-access-key-id <GOOGLE_ACCESS_KEY_ID> \
  --configuration-secret-access-key <GOOGLE_SECRET_ACCESS_KEY>

In the above command, replace

<MASTER_ENCRYPTION_KEY> with the Master Encryption Key from Step 2a.
<GOOGLE_CONFIGURATION_BUCKET> with your configuration bucket name.
<GOOGLE_ACCESS_KEY_ID> with your Google access key ID.
<GOOGLE_SECRET_ACCESS_KEY> with your Google secret access key.

Azure

storagefabric-gateway-setup \
  --gateway-domain s3.abccorp.com \
  --view default \
  --encrypted-view-encryption-key U2FsdGVkX18IR....S3P5/sM9ueS5L \
  --configuration-provider-url <AZURE_STORAGE_ACCOUNT>.blob.core.windows.net \
  --configuration-bucket azure-configuration-bucket \
  --configuration-access-key-id <AZURE_STORAGE_ACCOUNT> \
  --configuration-secret-access-key <AZURE_ACCESS_KEY> \
  --configuration-provider-api-type azure

In the above command, replace

<MASTER_ENCRYPTION_KEY> with the Master Encryption Key from Step 2a.
<AZURE_STORAGE_ACCOUNT> with your Azure storage account.
<AZURE_ACCESS_KEY> with your Azure container access key.

Note

If you want this setup to persist after your machine is rebooted, the –persist flag is required.

Note

If you make a mistake and want to run the storagefabric-gateway-setup again, the –reset-local flag is required to overwrite existing configuration.

Note

If S3 cloud services are accessible via a HTTP Proxy, use parameters (options) –http-proxy-host and –http-proxy-port to specify the proxy details.

After setting up the StorageFabric Gateway, exit from the storagefabric user

exit

Next Step: Configure StorageFabric

Configure StorageFabric

StorageFabric can be configured via the Command Line Interface (CLI), the StorageFabric Web Interface, or the StorageFabric Configuration Manager REST API.

In this tutorial, we will use either the CLI or web interface. For using the REST API, refer to the full product documentation.

Click on a link below to configure StorageFabric based on your preferred method.

Configure StorageFabric using the StorageFabric CLI
Configure StorageFabric using the Configuration Manager Web interface

Configure StorageFabric using the StorageFabric CLI

Add your Virtual Bucket, create Client Access Keys, grant client access to the Virtual Bucket, and sync configuration changes between the StorageFabric Configuration Manager and Gateway.

Note

Perform the following commands as the storagefabric user (sudo su storagefabric).

1. Add access keys for your Virtual Bucket to StorageFabric configuration.

Access keys for Virtual Buckets are also referred to as Provider Access Keys. Provider Access Keys are used by the StorageFabric Gateway to access the Virtual Bucket.

Amazon

storagefabric-configure add-cloud-access-key \
  --access-key-id <AWS_ACCESS_KEY_ID> \
  --secret-access-key <AWS_SECRET_ACCESS_KEY>

In the above command, replace

<AWS_ACCESS_KEY_ID> with your Amazon access key ID.
<AWS_SECRET_ACCESS_KEY> with your Amazon secret access key.

Google

storagefabric-configure add-cloud-access-key \
  --access-key-id <GOOGLE_ACCESS_KEY_ID> \
  --secret-access-key <GOOGLE_SECRET_ACCESS_KEY>

In the above command, replace

<GOOGLE_ACCESS_KEY_ID> with your Google access key ID.
<GOOGLE_SECRET_ACCESS_KEY> with your Google secret access key.

Azure

storagefabric-configure add-cloud-access-key \
  --access-key-id <AZURE_STORAGE_ACCOUNT> \
  --secret-access-key <AZURE_ACCESS_KEY>

In the above command, replace

<AZURE_STORAGE_ACCOUNT> with your Azure storage account.
<AZURE_ACCESS_KEY> with your Azure container access key.

2. Add your Virtual Bucket.

Amazon

storagefabric-configure add-bucket \
  --provider amazon \
  --views default \
  --cloud-access-key-id <AWS_ACCESS_KEY_ID> \
  --bucket virtual-data-bucket \
  --cloud-name <AWS_DATA_BUCKET>

In the above command, replace

<AWS_ACCESS_KEY_ID> with your Amazon access key ID.
<AWS_DATA_BUCKET> with an existing AWS data bucket name.

Google

storagefabric-configure add-bucket \
  --provider google \
  --views default \
  --cloud-access-key-id <GOOGLE_ACCESS_KEY_ID> \
  --bucket virtual-data-bucket \
  --cloud-name <GOOGLE_DATA_BUCKET>

In the above command, replace

<GOOGLE_ACCESS_KEY_ID> with your Google access key ID.
<GOOGLE_DATA_BUCKET> with an existing Google data bucket name.

Azure

storagefabric-configure --no-sync add-provider \
  --provider <AZURE_STORAGE_ACCOUNT> \
  --base-url <AZURE_STORAGE_ACCOUNT>.blob.core.windows.net \
  --api-type AZURE \
  --country US \
  --multipart-mode disabled \
  --tail-range-disable

storagefabric-configure add-bucket \
  --provider <AZURE_STORAGE_ACCOUNT> \
  --views default \
  --cloud-access-key-id <AZURE_ACCESS_KEY> \
  --bucket virtual-data-bucket \
  --cloud-name azure-data-bucket

In the above commands, replace

<AZURE_STORAGE_ACCOUNT> with your Azure storage account.
<AZURE_ACCESS_KEY> with your Azure container access key.

Note

In the above commands, --bucket specifies a Virtual Bucket Name, whereas --cloud-name is the actual bucket name with the backend provider.

cloud-name must refer to an existing backend bucket, this command does not create a new backend bucket.

Note

In this case, data encryption keys are automatically generated. KMS integrations are also possible.

3. Create access keys for Client.

Access keys used by clients to authenticate to the StorageFabric Gateway are referred to as Client Access Keys.

storagefabric-configure --no-sync create-client-access-key

New client access keys are generated and displayed as follows.

{
    "Code": "SUCCESS",
    "ClientAccessKeyID": "DC6550FF70876EBE5742",
    "ClientSecretAccessKey": "DYX81iqdWoUk0EJDX2G5XVt4uPCdTrcomZnaJPaq+l7Dx",
    "SuccessMessage": "Client Access Key created"
}

Save the client access key ID and secret access key generated for you, similar to the ones shown above. We will use them later to upload files securely via the StorageFabric Gateway.

4. Setup access control for Client Access Keys.

Grant the Client Access Key generated in step 3 read-write access to the Virtual Bucket.

storagefabric-configure add-role-to-client-access-key \
    --access-key-id <CLIENT_ACCESS_KEY_ID> \
    --role-name admin \
    --context '{ "providers" : "^.*$", "buckets" : "^.*$", "paths" : "^.*$" }'

In the above command, replace <CLIENT_ACCESS_KEY_ID> with the client access key ID generated in Step 3 above.

Note

In the above command, we assigned the built-in role admin to the client access key. For more information on creating and managing role-based access control, refer to the full product documentation.

5. Sync configuration changes between the StorageFabric Configuration Manager and Gateway.

storagefabric-sync --gateway

Note

Manual syncing is only for the purpose of this quickstart guide. In StorageFabric deployments, auto-syncing happens periodically in the background.

6. Revert to prior system user.

exit

Next Step: Upload files securely using StorageFabric

Configure StorageFabric using the Configuration Manager Web interface

Install and setup the Web Interface as described in the tutorial the full product documentation.

Once logged in to the Web Interface, continue to configure StorageFabric as follows.

1. Add access keys for your Virtual Bucket to StorageFabric configuration.

Access keys for Virtual Buckets are also referred to as Provider Access Keys. Provider Access Keys are used by the StorageFabric Gateway to access the Virtual Bucket.

Amazon

From the left-navigation bar, expand BACKENDS, OBJECT STORAGE and click Provider Access Keys.
Click the button Add Provider Access Key.
In the Access Key ID field, enter your <AWS_ACCESS_KEY_ID>.
In the Secret Access Key field, enter your <AWS_SECRET_ACCESS_KEY>.
If you are using temporary Provider Access Keys, also enter your Session Token in the Session Token field.
Leave the field Lifetime in minutes blank.
Check the option Admin Key. Checking this option means that StorageFabric can also use these credentials to create buckets with Amazon.
Click the button Add.

Google

From the left-navigation bar, expand BACKENDS, OBJECT STORAGE and click Provider Access Keys.
Click the button Add Provider Access Key.
In the Access Key ID field, enter your <GOOGLE_ACCESS_KEY_ID>.
In the Secret Access Key field, enter your <GOOGLE_SECRET_ACCESS_KEY>.
Leave the field Session Token blank.
Leave the field Lifetime in minutes blank.
Check the option Admin Key. Checking this option means that StorageFabric can also use these credentials to create buckets with Google.
Click the button Add.

Azure

From the left-navigation bar, expand BACKENDS, OBJECT STORAGE and click Provider Access Keys.
Click the button Add Provider Access Key.
In the Access Key ID field, enter your <AZURE_STORAGE_ACCOUNT>.
In the Secret Access Key field, enter your <AZURE_ACCESS_KEY>.
Leave the field Session Token blank.
Leave the field Lifetime in minutes blank.
Check the option Admin Key. Checking this option means that StorageFabric can also use these credentials to create containers with Azure.
Click the button Add.

2. Add your Virtual Bucket.

Link your Virtual Bucket with the Provider Access Key ID from Step 1.

Amazon

From the left-navigation bar, expand STORAGE and click Virtual Buckets.
Cick the button Create Virtual Bucket to open the Create Bucket form.
In the Virtual Bucket Name field, enter virtual-data-bucket. This is the Virtual Bucket Name.
In the Provider Name field, select amazon.
In the Backend Bucket Name field, enter the name of your AWS Backend Bucket. Note that this should be different than your Configuration Bucket. To automatically create the bucket with AWS, check the option Create Backend Bucket. If bucket is already created at AWS, uncheck the option Create Backend Bucket.
Select the Cloud Credentials tab.
In the Provider Access Key ID field, select your <AWS_ACCESS_KEY_ID>.
Click the button Create.

Google

From the left-navigation bar, expand STORAGE and click Virtual Buckets.
Cick the button Create Virtual Bucket to open the Create Bucket form.
In the Virtual Bucket Name field, enter virtual-data-bucket. This is the Virtual Bucket Name.
In the Provider Name field, select google.
In the Backend Bucket Name field, enter the name of your Google Backend Bucket. Note that this should be different than your Configuration Bucket. To automatically create the bucket with Google, check the option Create Backend Bucket. If bucket is already created at Google, uncheck the option Create Backend Bucket.
Select the Cloud Credentials tab.
In the Provider Access Key ID field, select your <GOOGLE_ACCESS_KEY_ID>.
Click the button Create.

Azure

From the left-navigation bar, expand BACKENDS and click Providers
Cick the button Add Provider to open the Add Provider form.
In the Provider Name field, enter your <AZURE_STORAGE_ACCOUNT>
In the Provider Base URL field, enter your <AZURE_STORAGE_ACCOUNT>.blob.core.windows.net
In the Country field, enter US.
Leave the remaining fields in the Connections tab as default.
Select the API Settings tab.
In the API Type field, select AZURE.
Uncheck the Tail Range Supported checkbox.
In the Multipart Mode field, select Disabled.
Click the button Add.

From the left-navigation bar, expand STORAGE and click Virtual Buckets.
Cick the button Create Virtual Bucket to open the Create Bucket form.
In the Virtual Bucket Name field, enter virtual-data-bucket. This is the Virtual Bucket Name.
In the Provider Name field, select your <AZURE_STORAGE_ACCOUNT>.
In the Backend Bucket Name field, enter your Azure container name. Note that this should be different than your Configuration Bucket. To automatically create the container with Azure, check the option Create Backend Bucket. If container is already created at Azure, uncheck the option Create Backend Bucket.
Select the Cloud Credentials tab.
In the Provider Access Key ID field, select your <AZURE_STORAGE_ACCOUNT>.
Click the button Create.

In this case, data encryption keys are automatically generated. KMS integrations are also possible.

3. Create access keys for Client.

Access keys used by clients to authenticate to the StorageFabric Gateway are referred to as Client Access Keys.

From the left-navigation bar, expand IDENTITY and click Client Access Keys.
Click the button Create Client Access Key to open the Create Client Access key form.
Click the button Create.

A new Access Key ID and Secret Access Key will be displayed. Copy and save them. We can use them to upload/download data to/from our Virtual Buckets.

New client access keys are generated and displayed at the top of the form.

Save the client access key ID and secret access key generate for you, similar to the ones shown above. We will use them later to upload files securely via the StorageFabric Gateway.

4. Sync configuration changes between the StorageFabric Configuration Manager and Gateway.

sudo su - storagefabric -c "storagefabric-sync --gateway"

Note

Manual syncing is only for the purpose of this quickstart guide. In StorageFabric deployments, auto-syncing happens periodically in the background.

Next Step: Upload files securely using StorageFabric

Upload files securely using StorageFabric

In this guide, we will use s3cmd as the client-side tool. Many other existing client-side tools can be used with StorageFabric. For more details, refer to the tutorial the full product documentation.

First, create a sample file using the command:

touch mydata.txt

Then, upload the sample file to your virtual-data-bucket using the following command with the Client Access Keys generated in step 3 earlier.

s3cmd put mydata.txt s3://virtual-data-bucket/ \
    --host localhost:8000 \
    --host-bucket localhost:8000 \
    --access_key <CLIENT_ACCESS_KEY_ID> \
    --secret_key <CLIENT_SECRET_ACCESS_KEY> \
    --signature-v2 --no-ssl

For remote clients, first add your StorageFabric Gateway Domain to clients’ /etc/hosts file or set up DNS for your StorageFabric Gateway Domain.

54.164.102.219   s3.abccorp.com

Then, use your StorageFabric Gateway Domain to access cloud storage services via the StorageFabric Gateway.

s3cmd put mydata.txt s3://virtual-data-bucket/ \
    --host s3.abccorp.com:8000 \
    --host-bucket s3.abccorp.com:8000 \
    --access_key <CLIENT_ACCESS_KEY_ID> \
    --secret_key <CLIENT_SECRET_ACCESS_KEY> \
    --signature-v2 --no-ssl

Note

To use SSL connections between clients and the StorageFabric Gateway, refer to the tutorial the full product documentation.