On November 20, 2020 Docker modified their policy by introducing pull rate limits for anonymous and free tier accounts. With this change in policy , the container image pull requests have been limited to 100 for anonymous users and 200 for free tier users every 6 hours. In addition to that – these limits are enforced on a per-manifest basis and as each image in Docker Hub may be a collection of manifests, pulling an image, even one that is cached, may involve a series of pull requests, thus exasperating the issue.

The installation of Astra Trident, NetApp’s open-source storage orchestrator for Kubernetes, requires that several images be pulled from Docker Hub as a part of the installation procedure. The introduction of rate limiting has resulted in many users encountering issues and receiving error messages such as: [“ERROR: toomanyrequests: Too Many Requests” OR “You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits”] when attempting to install Astra Trident on Red Hat OpenShift and other Kubernetes distributions.

To work around this issue, a user can register for a free account thus increasing their limit to 200 pull requests every six hours or they can register for a subscription-based account for a small monthly fee to greatly increase the allotment of pull requests. While either of these options will allow us to circumvent the rate limiting on image pulls, it now requires us to authenticate to Docker Hub while pulling the images we need to install Trident onto OpenShift. For its part OpenShift provides a convenient method to handle pull secrets at a cluster level, but by default, Docker login credentials are not added to it. This results in anonymous logins to the docker registry from the cluster and more often than not, users encounter the  pull request limit while fetching Trident images. Alternately, if it is not desired to handle pull secrets at a cluster level, Trident also provides a method to use a pull secret for it’s installation specifically.

Now that we understand the issue at hand and how to remedy it, let us explore how to configure Red Hat OpenShift to allow us to authenticate to Docker Hub and pull the images for Astra Trident, as we need them.

Fetch Docker Config

The first step would be to create an account on Docker Hub. You can either register as a free tier user or a paid user.  Once done, login to Docker Hub from any RHEL system with your user credentials –

For systems using podman:

# podman login docker.io

Username: tme_solutions

Password: ******

Login Succeeded!

 

This creates an authentication config file at ${XDG_RUNTIME_DIR}/containers/auth.json –

# cat ${XDG_RUNTIME_DIR}/containers/auth.json

{

        “auths”: {

                “docker.io”: {

                        “auth”: “a3Vsa2FybHdasjdleam46OTc5YTEyMzEtNTIwNy00ZasdsdmiqwlOaHRffiNWRh”

                }

        }

}

 

For systems using docker:

# docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.

 

Username: tme_solutions

Password: ******

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.

Login Succeeded!

 

This creates an authentication config file at /root/.docker/config.json –

# cat /root/.docker/config.json

{

        “auths”: {

                “https://index.docker.io/v2/”: {

                        “auth”: ” a3Vsa2FybHdasjdleam46OTc5YTEyMzEtNTIwNy00ZasdsdmiqwlOaHRffiNWRh”

                }

        },

        “HttpHeaders”: {

                “User-Agent”: “Docker-Client/19.03.13 (linux)”

        }

}

 

Global Configuration

OpenShift allows configuring pull secret for a registry at a cluster level which will be used by any container in any project to pull the images from that registry. Configuring docker secret at a cluster level will allow all pods in all the projects (not just Astra Trident) to pull docker images using the set credentials. For global configuration of docker credentials, it can either be passed to the installer while installing the OpenShift cluster (Greenfield) or updated after the OpenShift cluster is installed (Brownfield).

Greenfield Deployment

OpenShift installations require a pull secret to be provided to the installer for configuring the secrets for different image registries. Append the authentication details for Docker Hub to the pull secret that is passed on to the OpenShift installer.

From the authentication config file, copy the registry name and auth section. Append it to the pull secret you downloaded from Red Hat –

Content example to be appended:

“docker.io”: {“auth”: “a3Vsa2FybHdasjdleam46OTc5YTEyMzEtNTIwNy00ZasdsdmiqwlOaHRffiNWRh”}

Pull Secret example after appending auth details for Docker hub:

{“auths”:{“cloud.openshift.com”:{“auth”:”n6AHgy2Nlc3NfNDk1MDgyMDY5MzAzOWRiYzVhMDU6OTcwUlZQMzRJNUhTWTRZVDhUMDlZN0VGSUVVODZVNEJNRkY2Q0QyOTc5S0o3M000RUFIWjA5NjVFVjQyWTBVNw==”,”email”:”tme_solutions@netapp.com”},”quay.io”:{“auth”:”n6AHgy2Nlc3NfNDk1MDgyMDY5MzAzOWRiYzVhMDU6OTcwUlZQMzRJNUhTWTRZVDhUMDlZN0VGSUVVODZVNEJNRkY2Q0QyOTc5S0o3M000RUFIWjA5NjVFVjQyWTBVNw==”,””,”email”:”tme_solutions@netapp.com”},”registry.connect.redhat.com”:{“auth”:”fHVoYy1wb29sLTRlN2QwNTBmLWJlM2YtNDk4ZS1hZjgyLWI0MzFiZGNhMmRlNzpleUpoYkdjaU9pSlNVelV4TWlKOtBVGN6TktmUlZXcHcxOW9teEZwQ0lYZ1d3cjJobGxJeDBQN0xIZE0yeGM5Q0ZwZk5RR2JUanIxNnNUM21Rb0FJbUFjNC1BYlpEWVZEOHItNkxTMDZPUVpoWFRHcGwtRElDQ2RSYlJRaTlxbldLT2oyQ3pVeUJfNlIwcENSa2YyOUsyLWZGSFVfNA==”,”email”:”tme_solutions@netapp.com”},”registry.redhat.io”:{“auth”:”fHVoYy1wb29sLTRlN2QwNTBmLWJlM2YtNDk4ZS1hZjgyLWI0MzFiZGNhMmRlNzpleUpoYkdjaU9pSlNVelV4TWlKOtBVGN6TktmUlZXcHcxOW9teEZwQ0lYZ1d3cjJobGxJeDBQN0xIZE0yeGM5Q0ZwZk5RR2JUanIxNnNUM21Rb0FJbUFjNC1BYlpEWVZEOHItNkxTMDZPUVpoWFRHcGwtRElDQ2RSYlJRaTlxbldLT2oyQ3pVeUJfNlIwcENSa2YyOUsyLWZGSFVfNA==”:”tme_solutions@netapp.com”},”docker.io”:{“auth”:”a3Vsa2FybHdasjdleam46OTc5YTEyMzEtNTIwNy00ZasdsdmiqwlOaHRffiNWRh”}}}

Paste the modified pull secret while installing your OpenShift cluster. This ensures all the pull requests to Docker Hub from the cluster are through the account you created.

Brownfield Deployments

When you are trying to install Astra Trident on an existing OpenShift cluster, to mitigate the Docker ‘toomanyrequests’ issue, you will need to add the docker credentials to the existing pull secret and patch the global pull secret of the cluster.

Create a Docker config file:

cat << EOF > dockercfg.json

{“auths”:{“cloud.openshift.com”:{“auth”:”n6AHgy2Nlc3NfNDk1MDgyMDY5MzAzOWRiYzVhMDU6OTcwUlZQMzRJNUhTWTRZVDhUMDlZN0VGSUVVODZVNEJNRkY2Q0QyOTc5S0o3M000RUFIWjA5NjVFVjQyWTBVNw==”,”email”:”tme_solutions@netapp.com”},”quay.io”:{“auth”:”n6AHgy2Nlc3NfNDk1MDgyMDY5MzAzOWRiYzVhMDU6OTcwUlZQMzRJNUhTWTRZVDhUMDlZN0VGSUVVODZVNEJNRkY2Q0QyOTc5S0o3M000RUFIWjA5NjVFVjQyWTBVNw==”,””,”email”:”tme_solutions@netapp.com”},”registry.connect.redhat.com”:{“auth”:”fHVoYy1wb29sLTRlN2QwNTBmLWJlM2YtNDk4ZS1hZjgyLWI0MzFiZGNhMmRlNzpleUpoYkdjaU9pSlNVelV4TWlKOtBVGN6TktmUlZXcHcxOW9teEZwQ0lYZ1d3cjJobGxJeDBQN0xIZE0yeGM5Q0ZwZk5RR2JUanIxNnNUM21Rb0FJbUFjNC1BYlpEWVZEOHItNkxTMDZPUVpoWFRHcGwtRElDQ2RSYlJRaTlxbldLT2oyQ3pVeUJfNlIwcENSa2YyOUsyLWZGSFVfNA==”,”email”:”tme_solutions@netapp.com”},”registry.redhat.io”:{“auth”:”fHVoYy1wb29sLTRlN2QwNTBmLWJlM2YtNDk4ZS1hZjgyLWI0MzFiZGNhMmRlNzpleUpoYkdjaU9pSlNVelV4TWlKOtBVGN6TktmUlZXcHcxOW9teEZwQ0lYZ1d3cjJobGxJeDBQN0xIZE0yeGM5Q0ZwZk5RR2JUanIxNnNUM21Rb0FJbUFjNC1BYlpEWVZEOHItNkxTMDZPUVpoWFRHcGwtRElDQ2RSYlJRaTlxbldLT2oyQ3pVeUJfNlIwcENSa2YyOUsyLWZGSFVfNA==”:”tme_solutions@netapp.com”},”docker.io”:{“auth”:”a3Vsa2FybHdasjdleam46OTc5YTEyMzEtNTIwNy00ZasdsdmiqwlOaHRffiNWRh”}}}

EOF

Update global pull secret for Docker Hub for the entire cluster:

Patch the pull secret for the entire cluster –

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=dockercfg.json

This ensures all the pull requests to Docker Hub from the cluster are through the account you created.

NOTE: Updating the global pull secret causes the MCO to drain the nodes in a rolling fashion, reschedule the pods on other nodes, apply the change and then uncordon the nodes. This process might take some time depending on the size of the cluster.

Astra Trident specific Configuration

If the intent is just to ensure Astra Trident is installed without necessarily considering resolving the docker ‘toomanyrequests’ issue cluster-wide, you can create a secret with docker credentials and pass it to the Trident installer.

Create the namespace for Astra Trident:

oc create namespace trident

Create the secret for Docker Hub:

Create a Docker config file:

cat << EOF > dockercfg.json

{“auths”:{“docker.io”:{“auth”:”a3Vsa2FybHdasjdleam46OTc5YTEyMzEtNTIwNy00ZasdsdmiqwlOaHRffiNWRh”}}}

EOF

Create the secret for Docker Hub using the Docker config file:

oc create secret generic dockerconfig --from-file=.dockercfg=dockercfg.json --type=kubernetes.io/dockercfg -n trident

NOTE: Alternately, you can also create the secret for Docker Hub without creating Docker config file using a single command –

oc create secret docker-registry dockerconfig --docker-username=<username> --docker-password=<password> --docker-email=<email> --docker-server=https://index.docker.io/v2/ -n trident

Pass the pull secret to Astra Trident installer:

While installing Astra Trident, pass the pull secret to the installer by setting the imagePullSecrets parameter.

Install Astra Trident using the following helm command –

helm install trident trident-operator-21.04.0.tgz --namespace trident --set imagePullSecrets={dockerconfig}

This should ensure that the Trident pods use the appropriate Docker credentials to fetch the images from Docker Hub.


NOTE: If you are using a free tier account, the pull request limit is 200 for every 6 hours. If the Trident pods are failing with toomanyrequests error even after logging in with free tier Docker Hub account, you might have exceeded that limit for your account and might need to explore subscription-based options or login through another account.

Nikhil M Kulkarni
Technical Marketing Engineer at NetApp
Nikhil M Kulkarni is a Techincal Marketing Engineer at NetApp who focuses on architecting and validating Hybrid Cloud solutions and publishing NetApp Validated Architectures and Technical Reports, particularly on Container Orchestration and Automation.

Pin It on Pinterest