Note: This deployment method is not officially supported.
Overview
In this post, we will extract the Docker images from the Ubuntu/Debian bare metal distribution packages, make the images available to our Kubernetes worker nodes and deploy StorageGRID onto our Kubernetes cluster. Our example shows a minimal, single site StorageGRID deployment with one primary Admin Node and three Storage Nodes.
Networking
Kubernetes pods within the same cluster can communicate directly with each other; this is perfect for a single StorageGRID site as all the StorageGRID nodes need to intercommunicate. Additionally, the three Storage Nodes will register with the Admin Node by providing the Admin Node’s DNS name during deployment. A Kubernetes headless service will provide DNS mappings for the StorageGRID nodes. Using the DNS name, as opposed to the IP, will also allow the Storage Nodes to be deployed in parallel with the Admin Node.
Note: The existence of multiple sites would require StorageGRID nodes at each site be able to route directly to StorageGRID nodes at the other sites. This could be accomplished with a VPN or other tunneling technology and is outside the scope of this blog.
StorageGRID Nodes
Admin Node (one required)
A primary Admin Node is required for every StorageGRID deployment (secondary Admin Nodes are optional). Admin Nodes provide the Grid Manager Interface (GMI) for administering your StorageGRID deployment using a web browser.
Storage Node (three required)
Three Storage Nodes are required per StorageGRID site. Storage Nodes are the backbone of a StorageGRID deployment
Archive Node (optional)
Archive Nodes are optional. Archive Nodes act as an agent to a Tivoli Storage Manager (TSM) or Amazon S3 account for indefinite storage.
Gateway Node (optional)
The API Gateway Node monitors the health of the grid and the usage of each Storage Node. Since this is a minimalistic deployment, we will forego the Gateway Node’s health checks and use Kubernetes basic round-robin built-in load-balancer.
For further discussion on StorageGRID load balancing, see this Technical Report.
Prerequisites
Kubernetes
- Command line access to Master and Worker nodes of the cluster.
- Resources for four StorageGRID nodes (1 primary Admin Node, 3 Storage Nodes)
Each StorageGRID node will require:- 24Gi of memory (total of 96Gi)
- 8 CPUs (total of 32 CPUs)
- Persistent Volumes for the StorageGRID primary Admin Node:
- 100Gi mounted at /var/local
- 200Gi mounted at /var/local/mysql_ibdata
- 200Gi mounted at /var/local/audit/export
- Persistent Volumes for three StorageGRID Storage Nodes:
- 100Gi mounted at /var/local (per Storage Node)
- 4Ti mounted at /var/local/rangedb/0 (per Storage Node)
- 4Ti mounted at /var/local/rangedb/1 (per Storage Node)
- 4Ti mounted at /var/local/rangedb/2 (per Storage Node)
StorageGRID 11.1.1 (Ubuntu or Debian Platform tgz)
- Direct your browser to (requires registration):
https://mysupport.netapp.com/NOW/download/software/storagegrid_webscale/11.1.1/ - Click CONTINUE and accept the EULA
- Select and download StorageGRID-Webscale-11.1.1-DEB.tgz
This file will be downloaded: StorageGRID-Webscale-11.1.1-DEB-20180823.2111.d8ae314.tgz
Kubernetes Worker Nodes Preparation
Load the StorageGRID Docker images on each Kubernetes node.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
$ tar –C /tmp –xf StorageGRID–Webscale–11.1.1–DEB–20180823.2111.d8ae314.tgz
$ dpkg –x /tmp/StorageGRID–Webscale–11.1.1/debs/storagegrid–webscale–images–11–1–1_11.1.1–20180823.2111.d8ae314_amd64.deb /tmp/StorageGRID–Webscale–11.1.1/
$ sudo docker load –i /tmp/StorageGRID–Webscale–11.1.1/var/lib/storagegrid/images/11.1.1/storagegrid–11.1.1.tgz
Loaded image: storagegrid–11.1.1:Admin_Node
Loaded image: storagegrid–11.1.1:Storage_Node
Loaded image: storagegrid–11.1.1:Archive_Node
Loaded image: storagegrid–11.1.1:API_Gateway
$ sudo docker images storagegrid–11.1.1
REPOSITORY TAG IMAGE ID CREATED SIZE
storagegrid–11.1.1 Admin_Node b8158491dc8c 3 months ago 2.21GB
storagegrid–11.1.1 Storage_Node ef6e12967213 3 months ago 2.59GB
storagegrid–11.1.1 Archive_Node 9a103c2c07cf 3 months ago 1.81GB
storagegrid–11.1.1 API_Gateway c9eb2a6f3915 3 months ago 1.21GB
|
Note: Alternatively, the StorageGRID images could be hosted on a private Docker repository; just make sure to update the admin-node.yaml and the storage-node.yaml files with the image names.
Deploy StorageGRID on Kubernetes
Create a “storagegrid” namespace
File: storagegrid-namespace.yaml (raw)
|
1
2
|
$ kubectl create –f storagegrid–namespace.yaml
namespace “storagegrid” created
|
Create Persistent Volume Claims
This example uses Trident; however, any Kubernetes storage class can be used.
Note: Within this file, you must modify storageClassName to suit your Kubernetes system.
File: trident-persistent-volume-claims.yaml (raw)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
$ kubectl create ––namespace storagegrid –f trident–persistent–volume–claims.yaml
persistentvolumeclaim “var–local–dc1–adm–0″ created
persistentvolumeclaim “var–local–dc1–sn–0″ created
persistentvolumeclaim “var–local–dc1–sn–1″ created
persistentvolumeclaim “var–local–dc1–sn–2″ created
persistentvolumeclaim “mysql–dc1–adm–0″ created
persistentvolumeclaim “audit–dc1–adm–0″ created
persistentvolumeclaim “rangedb0–dc1–sn–0″ created
persistentvolumeclaim “rangedb1–dc1–sn–0″ created
persistentvolumeclaim “rangedb2–dc1–sn–0″ created
persistentvolumeclaim “rangedb0–dc1–sn–1″ created
persistentvolumeclaim “rangedb1–dc1–sn–1″ created
persistentvolumeclaim “rangedb2–dc1–sn–1″ created
persistentvolumeclaim “rangedb0–dc1–sn–2″ created
persistentvolumeclaim “rangedb1–dc1–sn–2″ created
persistentvolumeclaim “rangedb2–dc1–sn–2″ created
|
Wait until all the PVCs are bound before continuing.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
$ kubectl ––namespace storagegrid get pvc
NAME STATUS VOLUME CAPACITY
audit–dc1–adm–0 Bound storagegrid–audit–dc1–adm–0–ad3f7 200Gi
mysql–dc1–adm–0 Bound storagegrid–mysql–dc1–adm–0–ad364 200Gi
rangedb0–dc1–sn–0 Bound storagegrid–rangedb0–dc1–sn–0–ad4a0 4Ti
rangedb0–dc1–sn–1 Bound storagegrid–rangedb0–dc1–sn–1–ad67f 4Ti
rangedb0–dc1–sn–2 Bound storagegrid–rangedb0–dc1–sn–2–ad85f 4Ti
rangedb1–dc1–sn–0 Bound storagegrid–rangedb1–dc1–sn–0–ad53c 4Ti
rangedb1–dc1–sn–1 Bound storagegrid–rangedb1–dc1–sn–1–ad720 4Ti
rangedb1–dc1–sn–2 Bound storagegrid–rangedb1–dc1–sn–2–ad901 4Ti
rangedb2–dc1–sn–0 Bound storagegrid–rangedb2–dc1–sn–0–ad5d1 4Ti
rangedb2–dc1–sn–1 Bound storagegrid–rangedb2–dc1–sn–1–ad7ba 4Ti
rangedb2–dc1–sn–2 Bound storagegrid–rangedb2–dc1–sn–2–ad98c 4Ti
var–local–dc1–adm–0 Bound storagegrid–var–local–dc1–adm–0–ad0b5 100Gi
var–local–dc1–sn–0 Bound storagegrid–var–local–dc1–sn–0–ad169 100Gi
var–local–dc1–sn–1 Bound storagegrid–var–local–dc1–sn–1–ad207 100Gi
var–local–dc1–sn–2 Bound storagegrid–var–local–dc1–sn–2–ad2b6 100Gi
|
Create the headless DNS service
The headless DNS service will allow the use of a DNS name to reference the Admin Node.
File: pod-dns.yaml (raw)
|
1
2
|
$ kubectl create ––namespace storagegrid –f pod–dns.yaml
service “pod–dns” created
|
Deploy the Admin Node
File: admin-node.yaml (raw)
|
1
2
|
$ kubectl create ––namespace storagegrid –f admin–node.yaml
statefulset “dc1–adm” created
|
Verify the StatefullSet deployed and the pod started.
|
1
2
3
4
5
6
7
|
$ kubectl get ––namespace storagegrid statefulsets dc1–adm
NAME KIND
dc1–adm StatefulSet.v1.apps
$ kubectl get ––namespace storagegrid pod dc1–adm–0
NAME READY STATUS RESTARTS AGE
dc1–adm–0 1/1 Running 0 3m
|
Verify the Admin Node is waiting for configuration (last line in this output).
Note: This command only shows the last three lines.
|
1
2
3
4
|
$ kubectl logs ––namespace storagegrid dc1–adm–0 –tail 3
[INSG] Grid Manager has been started.
[INSG] Please direct your browser to http://10.112.2.100
[INSG] Waiting for configuration information
|
Deploy the Storage Nodes
File: storage-node.yaml (raw)
|
1
2
|
$ kubectl create ––namespace storagegrid –f storage–node.yaml
statefulset “dc1–sn” created
|
Verify the StatefulSet deployed and all the pods started.
|
1
2
3
4
5
6
7
8
9
|
$ kubectl get ––namespace storagegrid statefulsets dc1–sn
NAME KIND
dc1–sn StatefulSet.v1.apps
$ kubectl get ––namespace storagegrid pod dc1–sn–0 dc1–sn–1 dc1–sn–2
NAME READY STATUS RESTARTS AGE
dc1–sn–0 1/1 Running 0 2m
dc1–sn–1 1/1 Running 0 2m
dc1–sn–2 1/1 Running 0 2m
|
Verify that each Storage Node is waiting for approval (last line in this output – only dc1-sn-0 is shown).
Note: This command only shows the last three lines.
|
1
2
3
4
|
$ kubectl logs ––namespace storagegrid dc1–sn–0 –tail 3
[INSG]
[INSG]
[INSG] Please approve this node on the Admin Node GMI to proceed…
|
Deploy the Admin Node and Storage Nodes Services
- The Admin Node service will map GMI HTTPS (port 443) from the Admin Node to make it available at port 30443 of each Kubernetes node.
- The Storage Node service will map S3 protocol (port 18082) from the Storage Nodes to make them available at port 32182 of each Kubernetes node (Kubernetes will manage load-balancing).
File: storagegrid-services.yaml (raw)
|
1
2
3
|
$ kubectl create ––namespace storagegrid –f storagegrid–services.yaml
service “admin–node–servcie” created
service “storage–service” created
|
Verify the services started.
|
1
2
3
4
|
$ kubectl get ––namespace storagegrid service admin–node–servcie storage–service
NAME CLUSTER–IP EXTERNAL–IP PORT(S) AGE
admin–node–servcie 10.107.189.111 <nodes> 443:30443/TCP 1m
storage–service 10.99.231.197 <nodes> 18082:32182/TCP 1m
|
Install StorageGRID
- Direct your browser to https://<EXTERNAL-IP>:30443/
- Accept the default StorageGRID insecure certificate.
- From this point forward, you can follow the StorageGRID 11.1.1 installation documentation.
- Click Install a StorageGRID Webscale system.
Follow the installation wizard – paying special attention to:
- Step 1: License – Use the demo license that came with the download:
/tmp/StorageGRID-Webscale-11.1.1/debs/NLF000000.txt - Step 2: Grid Network – Use the network available to the Kubernetes pods
- Step 3: Grid Nodes – Make sure all four StorageGRID nodes get approved.
Note: NTP, DNS, and other fields will be specific to your environment, fill them in accordingly.
Click the Install button and monitor the installation.
After installation, simply:
- Log in as the administrator (use the password created during installation).
- Add Tenants and Buckets (making note of AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID).
- Don’t forget to direct your S3 client to port 32182 of the EXTERNAL-IP of the service.
Summary
At this point you have a fully functional StorageGRID system; albeit very minimum isolated single site grid. You can now leverage information lifecycle management (ILM) rules to manage object data.
Kubernetes and its deployment model have greatly simplified this whole process. We’re specifying compute resources, storage resources, DNS names, services to expose, etc.. Also, we’re deploying all the pods as StatefulSets, this makes all the PVCs stick their pods.
The six yaml files mentioned in this post can be found on GitHub. In addition, there are three more files enabling you to deploy an Archive Node (archive-node.yaml) and a Gateway Node (gateway-node.yaml, gateway-service.yaml).
Go kick the tires!!!