Kubernetes persistent volumes with NetApp Trident - Part 1

When you followed my blogs lately then you know that I recently installed Rancher locally in my homelab and deployed an RKE Kubernetes cluster on vSphere using Rancher with a simple Wordpress workload that makes use of persistent storage on vSphere. For this to happen, I first had to install and configure the vSphere CSI on Kubernetes using a Helm chart provided by Rancher. Although this is a valid configuration, the vSphere CSI has some disadvantages compared to making use of a CSI created by and for a specific storage solution like NetApp ONTAP which has snapshot and cloning capabilities.

Trident is NetApp’s open source storage orchestrator which integrates with Docker and Kubernetes, as well as platforms built on those technologies, such as Red Hat OpenShift, and Rancher. The goal of Trident is to make the provisioning, connection, and consumption of storage as transparent and friction less for containerized applications as possible. Trident automates storage management tasks for creation and provisioning against meta-classes of storage with common characteristics, providing abstracted pools of storage to the consumer. Users can take advantage of the underlying capabilities of NetApp storage infrastructure without having to know anything about it. 

In this two part blog I’ll explain how install and configure Trident for Kubernetes, define a NetApp ONTAP NFS backend, create a persistent volume and resize it. In the second part, I configure Kubernetes for storage snapshots, make a snapshot of a persistent volume, clone it and provision it to a new workload as you would do in a CI/CD pipeline.

So, let’s get started!

Install Trident

Trident is an out-of-tree storage provider and there are two ways you can deploy Trident.

  1. Deploying Trident with tridentctl
  2. Using the Trident operator

Using the Trident operator is the preferred way as the operator controls the installation of Trident, taking care to self-heal the install and manage changes as well as upgrades to the Trident installation. Starting with Trident 21.01, you can use helm to install Trident operator.

To start you need the kubeconfig file of your Kubernetes cluster. When you have Rancher and deployed an RKE cluster then this is very easy.

Login to Rancher, select your cluster. Then click on Kubeconfig File and copy it to your clipboard. Merge it with your existing local config file under ~/.kube/

Trident

Change the context to your cluster. Tip: with [kubectx](https://github.com/ahmetb/kubectx) you can switch very easy between multiple clusters!

Trident

Download the Trident installer.

wget https://github.com/NetApp/trident/releases/download/v21.01.2/trident-installer-21.01.2.tar.gz

tar -xf trident-installer-21.01.2.tar.gz

cd trident-installer

Trident

Create a new namespace for Trident and change the context to the trident namespace. Tip: with [kubens](https://github.com/ahmetb/kubectx) you can switch very easy between multiple namespaces!

kubectl create namespace trident

kubens trident

Trident

In the trident-installer go to the helm directory. Install the Trident operator with helm. Check if the trident operator and csi pods are running.

cd helm

helm install trident trident-operator-21.01.2.tgz

kubectl get pods

Trident

Trident

Use tridentctl to check the installed version.

cd trident-installer

./tridentctl version

Trident

Create a NFS backend for Trident

With Trident installed it now needs to know where to create volumes. This information sits in objects called backends. A backend contains:

  • The driver type
    • Currently there are 12 different drivers available within Trident for the various NetApp portfolio of on-prem and Cloud based storage
  • How to connect to the driver
    • SVM, IP address, username, password etc.
  • And some default parameters
    • Auto export policy, aggregate/volume usage limits etc.

Starting with Trident 21.04 backends can be created and managed directly through the Kubernetes interface using command-line tools such as kubectl. In my case I installed Trident 21.01.2 and have to create a backend using tridentctl.

If you created a backend using tridentctl and then upgrade to a newer version of trident (21.04+) the backend created with tridentctl will continue to remain operational throughout the upgrade. Afterwards you can create a TridentBackendConfig to manage pre-existing backends using kubectl if you so desire.

First, login to NetApp ONTAP Sytem Manager and get the following details:

  • SVM name
  • SVM Management LIF IP
  • SVM Data LIF IP for NFS
  • SVM admin username and password

Trident

Create a json file for the backend. For example, backend-ontap-nas.json

{
  "version": 1,
  "storageDriverName": "ontap-nas",
  "backendName": "ontap-nas-auto-export",
  "managementLIF": "10.0.0.143",
  "dataLIF": "10.0.0.144",
  "svm": "svm1",
  "username": "vsadmin",
  "password": "Netapp1!",
  "limitVolumeSize": "15Gi",
  "autoExportCIDRs": ["10.0.0.0/24"],
  "autoExportPolicy": true
}

In this example I use the Trident ontap-nas driver, connect to svm1 and limit the size of the a volume which can be created. I also enable autoExportPolicy for dynamic export policy management.

For additional information, please refer to the official NetApp Trident documentation on Read the Docs:

https://netapp-trident.readthedocs.io/en/latest/kubernetes/operations/tasks/backends/ontap/ontap-nas/index.html

Create the backend using tridentctl.

./tridentctl -n trident create backend -f backend-ontap-nas.json

Trident

Create a Kubernetes Storage Class

For users to create Persistent Volume Claims (PVC) in Kubernetes, one or more Storage Classes are needed. A storage class points to a backend and contains the definition of what an app can expect in terms of storage, defined by some properties (access type, media, driver etc).

For additional information, please refer to:

Create a yaml file for the Storage Class. For example, sc-csi-ontap-nas.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ontap-nas-auto-export
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
allowVolumeExpansion: true

Notice that I added the allowVolumeExpansion parameter. Setting this on true makes it possible to resize volumes after creation.

Create the Storage Class using kubectl

kubectl create -f sc-csi-ontap-nas.yaml

kubectl get -n trident tridentbackends

kubectl get sc

Trident

Also in Rancher the Storage Class is visible.

Trident

Deploy an app with persistent file storage

For this task I will be deploying Ghost (a light weight web portal) utlilizing RWX (Read Write Many) file-based persistent storage over NFS in three steps:

  1. Create a Persistent Volume Claim (PVC) for blog content
  2. Create a deployment for Ghost using the blog content PVC
  3. Create a service to expose the Ghost application

Create a new directory for the Ghost app. Within this directory create three files:

  • pvc.yaml
  • deploy.yaml
  • service.yaml

Edit the pvc.yaml file. Create a ghost namespace and the PVC using kubectl

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: blog-content
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: ontap-nas-auto-export
kubectl create namespace ghost

kubectl create -n ghost -f pvc.yaml

Trident

If we now login to ONTAP System Manager we should see a new Trident volume being created and automatically mounted, via the dynamic managed export policies, to the nodes of the Kubernetes cluster.

Trident

Trident

Edit the deploy.yaml file and deploy the Ghost app using kubectl

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog
  labels:
    app: blog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
      - name: blog
        image: ghost:2.6-alpine
        imagePullPolicy: Always
        ports:
        - containerPort: 2368
        env:
        - name: url
          value: http://my-blog.homelab.int
        volumeMounts:
        - mountPath: /var/lib/ghost/content
          name: content
      volumes:
      - name: content
        persistentVolumeClaim:
          claimName: blog-content
kubectl create -n ghost -f deploy.yaml

Trident

Edit the service.yaml file and expose the Ghost app using kubectl

apiVersion: v1
kind: Service
metadata:
  name: blog
spec:
  #if you have a load balancer installed use type: LoadBalancer
  type: NodePort
  selector:
    app: blog
  ports:
  - protocol: TCP
    port: 80
    targetPort: 2368
    nodePort: 30080
kubectl create -n ghost -f service.yaml

Trident

Let’s go over to Rancher and see if the workload and it’s persistent volume are visible.

First, move the ghost namespace to a project. In my case the homelab project.

Select the cluster. In the top menu click Projects/Namespaces. Select ghost under Namespace: all and click on the submenu (3 dots) to move it to a project.

TridentSelect your project. In my case, homelab. Click Move.

Trident

Click on the project and select the Workloads tab. Under Namespace: ghost, the blog app is active.

Trident

Select the Volumes tab to show the persistent volumes (PV) and claims (PVC). Under Namespace: ghost, you should see the PVC blog-content and the related PV being bound.

Trident

This means the Ghost app is up-and-running. Let’s check and create some content.

Depending on if you have used the LoadBalancer or NodePort type in your service configuration, go to http://YOUR_EXTERNAL_IP/admin or http://YOUR_WORKERNODE_IP:30080/admin

Create a Ghost admin account by clicking Create your account.

Trident

Enter your account details and click Last step: Invite your team.

TridentSkip this by clicking I’ll do this later, take me to my blog.

Create a new story and publish it.

Trident

Logout and point your web browser to http://YOUR_EXTERNAL_IP or http://YOUR_WORKERNODE_IP:30080

Trident

Confirm data persistence

To confirm data persistence we restart the blog pod on another node by draining the worker node where it’s running on.

First check if all cluster nodes are up and on which worker node the blog pod is running.

kubectl get nodes -o wide

kubectl get pod -l app=blog -n ghost -o wide

In my case the blog pod is running on node cls01-worker2. Let’s drain the node and set it in maintenance mode.

kubectl drain cls01-worker2 --ignore-daemonsets

All running pods, daemonsets excluded, are now being stopped and rescheduled on the remaining worker node(s).

Trident

Check if node cls01-worker2 is in maintenance mode and if the blog pod is now running on another worker node.

kubectl get nodes

kubectl get pod -l app=blog -n ghost -o wide

Trident

Refresh your web browser and access your blog post to confirm it still exists.

Resize the volume

Trident supports the resizing of File (NFS/RWX) & Block (iSCSI/RWO) PVC’s, depending on the Kubernetes version.
NFS resizing was introduced in Kubernetes 1.11, while iSCSI resizing was introduced in Kubernetes 1.16. Resizing a PVC is made available through the option allowVolumeExpansion set in the Storage Class.

To resize a persistent volume you have to edit the PVC object.

kubectl get pvc -n ghost

kubectl edit pvc blog-content -n ghost
#under spec change the required size

spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

Trident

When I configured the NFS backend in Trident, I set the limitVolumeSize to 15Gi. So, what happens if I try to create a PVC that’s bigger than 15Gi?

Create a pvc_tobig.yaml file and try to create a new PVC using kubectl

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: big-blog-content
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: ontap-nas-auto-export
kubectl create -n ghost -f pvc_tobig.yaml

kubectl get pvc -n ghost

Trident

The new PVC object is in pending state.

Login to Rancher and investigate why the PVC in in pending state.

Select the cluster. In the top menu click on Storage. In the drop down menu select Persistent Volumes.

TridentClick on the big-blog-content PVC in pending state and expand Events.

TridentUnder Events you see the Warning Provisioning Failed. In the Message you can see that the requested size was bigger than the size limit. Due to this the PVC will not be provisioned. Mission accomplished!

Next will be part 2 where I’ll explain how to enable storage snapshots & cloning in Kubernetes, make a clone of a persistent volume and use it for testing an upgrade of the Ghost blog app.