Red Hat's OpenShift is one of the more popular Platform-as-a-Service offerings for deploying containerized applications. It's built on Kubernetes, but also adds a large number of additional services to make the process of creating, deploying, and managing applications easy.

One of the services which OpenShift provides is a version of the Docker Registry to host images for things like build deployments and local caching. Like any Registry instance, it can use a number of different types of storage, depending on what you have available and what your overall goals are. For our purposes, this includes:

  • An NFS export from an ONTAP system
  • An iSCSI LUN from an ONTAP or SolidFire system
  • An S3 (or Swift) bucket from StorageGRID WebScale

Each of these has some advantages and disadvantages, however we won't discuss those in this post. Instead, we're going to focus on how to connect to the different types of storage with the OpenShift deployed Registry instance. Before we begin, let's look at some pre-requisites:

  • OpenShift has already been deployed. This blog post was created using OpenShift 3.5.
  • Your storage is available:
    • Trident has been deployed or
    • A StorageGRID account with a bucket has been created

The process of modifying the Registry to use different storage follows just a few steps:

  1. If not already deployed, deploy the Registry
  2. Assess the existing storage configuration
  3. Provision new storage
  4. Update the Registry configuration
  5. Redeploy with the new configuration

Deploying the Registry

Before we begin, let's discover the current status of the Registry for your OpenShift deployment. Many times the Registry will be deployed by default.

The easiest way to determine if the Registry is currently deployed is to do a dry run:

# login as the system admin
oc login -u system:admin

# ensure we're in the default project
oc project default

# check the status
oc adm registry --dry-run

This will result in a message stating the service either does or does not exist. Here is the message when the service does exist:

Checking if the Registry has been deployed

If you do not have the Registry already deployed, then before we can modify the configuration of the Registry, we need to have it deployed using the default configuration. This is very simple:

# ensure we are logged in and using the correct project
oc login -u system:admin && oc project default

# deploy the registry
oc adm registry

This will create an instance of the Registry with a single replica, which will act as the basis for our future modifications.

Determining the Current Configuration

With the Registry deployed, let's inspect the configuration.

# ensure we're logged in as the correct user in the correct project
oc login -u system:admin && oc project default

# view the deploymentconfig for the registry
oc describe dc docker-registry

At this point, we should see a lot of information about the Registry, however we're really only interested in one section: the volumes.

Volumes associated with the Registry

In this instance there are three volumes defined:

  1. A PersistentVolumeClaim (PVC) named "registry-storage". We can get more information about this PVC using the command oc describe pvc registry-storage.
  2. A Secret named "registry-certificates". As the name infers, this contains the SSL certificates being used by the Registry instance.
  3. A Secret named "registry-config". We can infer that this refers to a customized Registry configuration file, we'll look at how to confirm that below.

To get more information about each one, we want to use the oc volume command.

Volume details for the Registry deployment config

We can now see details about where each volume is being used.

  1. The PVC registry-storage is mounted at /registry in the container
  2. The secret registry-certificates is mounted at /etc/secrets
  3. The secret registry-config is mounted at /etc/docker/registry

/registry is the default location for the Registry storage when using the filesystem driver, this could be where data is being stored, however also notice that there is another volume which appears to be overriding the configuration file. We can infer this because the config file is stored in that directory. If we want to confirm that the file /etc/docker/registry/config.yml is being supplied by that secret, we can get the secret using the command oc get secret registry-config -o yaml, which, as we see below contains a file named "config.yml", confirming that it contains a customized configuration file.

Viewing the contents of the secret

To view the contents, the easiest method is to simply extract the file from the container:

# ensure we are the correct user in the correct project
oc login -u system:admin && oc project default

# find a registry pod, note there may be more than one if
# your Registry has been scaled beyond one replica
oc get pod | grep docker-registry | awk '{ print $1 }'

# retrieve the file
oc exec -it docker-registry-- cat /etc/docker/registry/config.yml > config.yml

# show the file
cat config.yml

Note that "<deployment id>" and "<unique id>" will be different for your environment.

Earlier I had stated that having a volume mounted at /registry may be an indicator that the specified PVC is being used for Registry storage, but was not a sure thing. This is because the config.yml can change this location to a different file system location, or even configure different storage all together, for example S3 or Azure Blob storage.

After assessing the current configuration to determine the storage configuration, we have all of the information we need and are ready to move on. Most likely you will have one of two configurations:

  • The Registry is using the filesystem driver with local storage.
  • The Registry is using the filesystem driver with a PVC, which could be from a number of places. For example, the OpenShift VMware Reference Architecture will deploy a virtual machine to act as an NFS target for the Registry storage. To determine the backing storage for the PVC, use the command oc describe pv registry-storage, which will display the type of storage and where it's being sourced from.

Red Hat recommends that storage more robust than what's provided with the default configuration be used. Using storage provided by a NetApp platform, such as StorageGRID, ONTAP, SolidFire, or E-Series provides a significant increase in performance, availability, and simplicity for things like backup and recovery of the Registry data. In the next sections we'll cover how to use the NetApp storage platforms with the OpenShift Registry.

OpenShift Registry using StorageGRID WebScale

Object storage is an excellent choice for Registry storage, and is generally the default recommendation from Docker for the most scalable deployment. The first step when using object storage is to extract the current configuration file being used by the Registry. To do that, we will follow the same steps as above:

# ensure we are the correct user in the correct project
oc login -u system:admin && oc project default

# find a registry pod, note there may be more than one if
# your Registry has been scaled beyond one replica
oc get pod | grep docker-registry | awk '{ print $1 }'

# retrieve the file
oc exec -it docker-registry-- cat /etc/docker/registry/config.yml > config.yml

# edit the file
vi config.yml

We need to replace the storage section with our StorageGRID credentials. In this instance, I'm using the S3 interface:

    region: local
    regionendpoint: https://:

With the config.yml updated, we need to replace the existing configuration file for the Registry.

# ensure we are logged in correctly and using the default project
oc login -u system:admin && oc project default

# check for an existing secret, delete if necessary, note that this
# secret name could be different, be sure it matches your config
oc get secret registry-config && oc delete secrete registry-config

# create a new secret with the contents of the configuration file
oc secret new registry-config config.yml=./config.yml

# add the secret as a volume to the deployment config
oc volume dc docker-registry --add --type=secret \
  --secret-name=registry-config \
  -m /etc/docker/registry/ \

# create an environment variable for Registry to use the modified
# config file
oc set env dc docker-registry \

At this point we're ready to redeploy the Registry with the new configuration, skip down to that section to see how to redeploy the Registry using the new settings.

OpenShift Registry using iSCSI or NFS

To use iSCSI or NFS we first need to create a PVC in the default namespace. Since we are using Trident to do the storage provisioning, we also want to make sure that Trident has been deployed and configured with at least one backend.

This process is exactly the same regardless of whether you are using ONTAP NFS, ONTAP iSCSI, or SolidFire iSCSI. However be aware that using iSCSI will limit you to just a single replica of the Registry. This is because each replica expects to have shared access to the same storage, which doesn't work with block storage protocols. If you want to have more than one replica, use NFS (or object, covered below) storage.

# Create a PVC using a storage class.  Note that this PVC has the
# beta storage class specification.  If you are using OpenShift <= 3.5
# this is mandatory.  If you are using OpenShift >= 3.6 this will
# work, but should be updated to use the v1 specification.
cat << EOF > pvc-registry.yaml
kind: PersistentVolumeClaim
apiVersion: v1
  name: trident-registry-storage
  annotations: <storage class name>
    - ReadWriteMany
      storage: 500Gi

# submit the PVC
oc create -f pvc-registry.yaml

# update the volume config to use our new PVC
oc volume dc docker-registry --add --name=registry-storage -t pvc --claim-name=trident-registry-storage --overwrite

At this point, we're done with the configuration changes, proceed to the next section to redeploy the registry with the new settings.

Redeploying the Registry After a Change

Once the storage and configuration have been updated to your needs, the Registry needs to be redeployed with the new configuration. The Registry is managed as a deployment, which makes it very easy to safely redeploy an application.

To view the current deployments, and their status, use the command oc get dc. To redeploy using the latest configuration, use the oc rollout latest docker-registry command.

Checking the status of the rollout

Monitor the progress using the oc get dc command. After a moment, you should see the revision increase along with the current value reflecting the number of currently deployed pods.

Verifying the New Storage

Finally, to test functionality, let's connect directly to the Registry and push an image. Follow OpenShift's instructions for configuring your user for direct access to the Registry if needed. To determine the address of the internal Registry, use the oc route command.

# display the external name for the Registry
oc get route docker-registry

For my OpenShift cluster, this is docker-registry-default.apps.vtme.local.

# login as a regular user and associate with a project
oc login -u  && oc project 

# pull an image
docker pull alpine

# retag for the local registry
docker tag alpine docker-registry-default.//alpine

# push the image
docker push docker-registry-default.//alpine

Verifying the newly configured Registry

To be thorough, and confident, in validating the newly configured registry, let's remove the local image and pull it from the OpenShift Registry. This validates that we can both push and pull images after our configuration change.

# remove the existing image
docker rmi docker-registry-default.//alpine

# pull the image from the OpenShift Registry
docker pull docker-registry-default.//alpine

Assuming your image was successfully pulled, you're done! If you encounter issues use the standard troubleshooting methods for OpenShift applications to determine where the error may be. This includes checking the pods (oc get pod) and their logs (oc logs) if the containers are running, or doing a describe (oc describe pod) if unsuccessful.


Using robust storage for the Registry is important to providing a high quality user experience with OpenShift, or any deployment where you want to store private or customized container images. Integrating NetApp's StorageGRID WebScale, or NFS/iSCSI storage using Trident, makes it extremely simple to provide all of the features and capabilities which users expect from the Regsitry. This process only takes a few minutes from start to finish, and, when leveraging storage from the NetApp portfolio, adds enterprise reliability and performance for your Registry instance.

If you have any questions, please leave a comment below or reach out using our Slack channels.

About Andrew Sullivan

Andrew has worked in the information technology industry for over 10 years, with a rich history of database development, DevOps experience, and virtualization. He is currently focused on storage and virtualization automation, and driving simplicity into everyday workflows.

Pin It on Pinterest