NetApp’s StorageGRID object store is very easy to use, but it can be challenging to get all of the right infrastructure together to deploy it, especially if you just want to try it out. But under the covers it’s built and deployed as container images, and recent Kubernetes-friendly enhancements make it possible to use those to deploy StorageGRID in an existing Kubernetes cluster, making it easy to try as well!

Note: This deployment method is not officially supported.

Overview

In this post, we will extract the Docker images from the Ubuntu/Debian bare metal distribution packages, make the images available to our Kubernetes worker nodes and deploy StorageGRID onto our Kubernetes cluster. Our example shows a minimal, single site StorageGRID deployment with one primary Admin Node and three Storage Nodes.

Networking
Kubernetes pods within the same cluster can communicate directly with each other; this is perfect for a single StorageGRID site as all the StorageGRID nodes need to intercommunicate.  Additionally, the three Storage Nodes will register with the Admin Node by providing the Admin Node’s DNS name during deployment.  A Kubernetes headless service will provide DNS mappings for the StorageGRID nodes.  Using the DNS name, as opposed to the IP, will also allow the Storage Nodes to be deployed in parallel with the Admin Node.

Note: The existence of multiple sites would require StorageGRID nodes at each site be able to route directly to StorageGRID nodes at the other sites.  This could be accomplished with a VPN or other tunneling technology and is outside the scope of this blog.

StorageGRID Nodes

Admin Node (one required)
A primary Admin Node is required for every StorageGRID deployment (secondary Admin Nodes are optional).  Admin Nodes provide the Grid Manager Interface (GMI) for administering your StorageGRID deployment using a web browser.

Storage Node (three required)
Three Storage Nodes are required per StorageGRID site.  Storage Nodes are the backbone of a StorageGRID deployment

Archive Node (optional)
Archive Nodes are optional.  Archive Nodes act as an agent to a Tivoli Storage Manager (TSM) or Amazon S3 account for indefinite storage.

Gateway Node (optional)
The API Gateway Node monitors the health of the grid and the usage of each Storage Node.  Since this is a minimalistic deployment, we will forego the Gateway Node’s health checks and use Kubernetes basic round-robin built-in load-balancer.

For further discussion on StorageGRID load balancing, see this Technical Report.

Prerequisites

Kubernetes

  • Command line access to Master and Worker nodes of the cluster.
  • Resources for four StorageGRID nodes (1 primary Admin Node, 3 Storage Nodes)
    Each StorageGRID node will require:

    • 24Gi of memory (total of 96Gi)
    • 8 CPUs (total of 32 CPUs)
  • Persistent Volumes for the StorageGRID primary Admin Node:
    • 100Gi mounted at /var/local
    • 200Gi mounted at /var/local/mysql_ibdata
    • 200Gi mounted at /var/local/audit/export
  • Persistent Volumes for three StorageGRID Storage Nodes:
    • 100Gi mounted at /var/local (per Storage Node)
    • 4Ti mounted at /var/local/rangedb/0 (per Storage Node)
    • 4Ti mounted at /var/local/rangedb/1 (per Storage Node)
    • 4Ti mounted at /var/local/rangedb/2 (per Storage Node)

StorageGRID 11.2.0 (Ubuntu or Debian Platform tgz)

Kubernetes Worker Nodes Preparation

Load the StorageGRID Docker images on each Kubernetes node.

Note: Alternatively, the StorageGRID images could be hosted on a private Docker repository; just make sure to update the admin-node.yaml and the storage-node.yaml files with the image names.

Deploy StorageGRID on Kubernetes

Create a “storagegrid” namespace

File: storagegrid-namespace.yaml (raw)

Create Persistent Volume Claims

This example uses Trident; however, any Kubernetes storage class can be used.

Note: Within this file, you must modify storageClassName to suit your Kubernetes system.

File: trident-persistent-volume-claims.yaml (raw)

Wait until all the PVCs are bound before continuing.

Create the headless DNS service

The headless DNS service will allow the use of a DNS name to reference the Admin Node.

File: pod-dns.yaml (raw)

Deploy the Admin Node

File: admin-node.yaml (raw)

Verify the StatefullSet deployed and the pod started.

Verify the Admin Node is waiting for configuration (last line in this output).

Note: This command only shows the last three lines.

Deploy the Storage Nodes

File: storage-node.yaml (raw)

Verify the StatefulSet deployed and all the pods started.

Verify that each Storage Node is waiting for approval (last line in this output – only dc1-sn-0 is shown).

Note: This command only shows the last three lines.

Deploy the Admin Node and Storage Nodes Services

  • The Admin Node service will map GMI HTTPS (port 443) from the Admin Node to make it available at port 30443 of each Kubernetes node.
  • The Storage Node service will map S3 protocol (port 18082) from the Storage Nodes to make them available at port 32182 of each Kubernetes node (Kubernetes will manage load-balancing).

File: storagegrid-services.yaml (raw)

Verify the services started.

Install StorageGRID

  • Direct your browser to https://:30443/
  • Accept the default StorageGRID insecure certificate.
  • From this point forward, you can follow the StorageGRID 11.2.0 installation documentation.
  • Click Install a StorageGRID Webscale system.

Follow the installation wizard – paying special attention to:

  • Step 1: License – Use the demo license that came with the download:
    /tmp/StorageGRID-Webscale-11.2.0/debs/NLF000000.txt
  • Step 2: Grid Network – Use the network available to the Kubernetes pods
  • Step 3: Grid Nodes – Make sure all four StorageGRID nodes get approved.

Note: NTP, DNS, and other fields will be specific to your environment, fill them in accordingly.

Click the Install button and monitor the installation.

After installation, simply:

  • Log in as the administrator (use the password created during installation).
  • Add Tenants and Buckets (making note of AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID).
  • Don’t forget to direct your S3 client to port 32182 of the EXTERNAL-IP of the service.

Summary

At this point you have a fully functional StorageGRID system; albeit very minimum isolated single site grid. You can now leverage information lifecycle management (ILM) rules to manage object data.

Kubernetes and its deployment model have greatly simplified this whole process. We’re specifying compute resources, storage resources, DNS names, services to expose, etc.. Also, we’re deploying all the pods as StatefulSets, this makes all the PVCs stick their pods.

The six yaml files mentioned in this post can be found on GitHub. In addition, there are three more files enabling you to deploy an Archive Node (archive-node.yaml) and a Gateway Node (gateway-node.yamlgateway-service.yaml).

Go kick the tires!!!

Mauricio Sánchez
StorageGRID platform developer at NetApp Inc.
27 years in the making, fresh, zingy and hand-selected with a mature elegance
• Architects, deploys and manages enterprise application platforms
• Automates everything: Builds, Deployments, Process
• Improves Engineering Productivity and SW Development tools

Pin It on Pinterest