In case you hadn’t noticed, the 20.01 release delivered support for Prometheus metrics. Trident now exposes a host of metrics that can be queried with Prometheus and used to obtain greater insight into the operations that Trident handles. This blog walks you through configuring Trident 20.01.1 and Prometheus, providing a great way to view metrics on Trident’s performance.

 

My Setup

I am working on a Kubernetes cluster that I had set up with kubeadm. My cluster runs Kubernetes 1.17.

$ kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
node1   Ready    master   9m14s   v1.17.2
node2   Ready    <none>   8m48s   v1.17.2
node3   Ready    <none>   8m42s   v1.17.2

I will go ahead and install Prometheus on this cluster using Helm. To install Prometheus, I am using the prometheus-operator chart. You can get the helm binary from the GitHub repo. I am using Helm v3.0.3, which is the most stable release available right now.

cd /tmp
wget https://get.helm.sh/helm-v3.0.3-linux-amd64.tar.gz
tar xzvf helm-v3.0.3-linux-amd64.tar.gz
cp linux-amd64/helm /usr/bin/

 

I install the Prometheus-operator chart. It is as easy as:

$ kubectl create namespace monitoring
namespace/monitoring created
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com
$ helm repo update
$ helm install prom-operator stable/prometheus-operator --namespace monitoring 

 

This should setup the Prometheus operator in its own namespace [called monitoring]. You should be able to see a list of services, serviceMonitors and other objects that have been deployed using this operator.

 

Install Trident and create a ServiceMonitor

I proceed to install Trident 20.01.1 on my cluster. Follow our installation guide to explore the different options for installing Trident. It’s usually as simple as retrieving the installer and running tridentctl install.

$ tridentctl version -n trident
+---------------------------------------------------------+---------------------------------------------------------+
|                     SERVER VERSION                      |                     CLIENT VERSION                      |
+---------------------------------------------------------+---------------------------------------------------------+
| 20.01.0-custom+ef519bceb6e82e948d56a8f1acee5d801ae23637 | 20.01.0-custom+ef519bceb6e82e948d56a8f1acee5d801ae23637 |
+---------------------------------------------------------+---------------------------------------------------------+

 

For Prometheus to retrieve the metrics that Trident exposes, I will need to create a ServiceMonitor. Prometheus introduces the ServiceMonitor CRD to define which services need to monitored, using labels. You could define a ServiceMonitor to watch over several services by choosing the labels to monitor. Here, I define a ServiceMonitor to watch the trident-csi service that’s automatically set up when Trident is installed:

#####servicemonitor.yaml#####
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: trident-sm
  namespace: monitoring
  labels:
    release: prom-operator
spec:
  jobLabel: trident
  selector:
    matchLabels:
      app: controller.csi.trident.netapp.io 
  namespaceSelector:
    matchNames:
    - trident
  endpoints:
  - port: metrics
    interval: 15s

The ServiceMonitor does a couple of things:

  1. It looks for metrics retrieved by the trident-csi service (look for the “app” label that we match. This “app” label is present on the trident-csi service running in the “trident” namespace, specified in the namespaceSelector).
  2. The endpoint for these metrics is defined to be the metrics port that the trident-csi service exposes.

You must make sure you include the release label for the ServiceMonitor when you define it; the prometheus-operator creates a Prometheus object which only looks for ServiceMonitors with the release label set to your Helm release name [look for the spec.serviceMonitorSelector in your Prometheus object]

Now I can expose the Prometheus dashboard by forwarding 9090. Here I am using kubectl to forward 9090 but you can also create a NodePort Service to expose the dashboard:

kubectl port-forward -n monitoring prometheus-prom-operator-prometheus-0 9090

Trident’s ServiceMonitor in the list of Prometheus targets

I can take a look at all the endpoints that Trident provides and use them to generate graphs from the Prometheus dashboard:

Trident’s endpoints

Using the graph feature in Prometheus, you can define simple queries to pull Trident’s metrics.

Prometheus graph for Trident rest ops

You can also use Grafana to create a more visual and descriptive graph. Grafana is setup by the Prometheus-operator by default. You can use the Prometheus dashboard within Grafana to set up a datasource that pulls Trident’s metrics like this:

A Grafana dashboard that displays Trident’s Prometheus endpoints

And there you have it. A convenient way to monitor Trident from within Kubernetes, using Prometheus and Grafana.

Stay in touch

Join us in our Slack workspace in the #containers channel.

Bala RameshBabu
Bala is a Technical Marketing Engineer who focuses on Trident, NetApp's dynamic storage provisioner for Kubernetes and Docker. With a background in OpenStack, he focuses on open-source solutions and DevOps workflows. When not at work, you can find him in a soccer field or reading biographies

Pin It on Pinterest