Kubernetes persistent volumes with NetApp Trident - Part 1
When you followed my blogs lately then you know that I recently installed Rancher locally in my homelab and deployed an RKE Kubernetes cluster on vSphere using Rancher with a simple Wordpress workload that makes use of persistent storage on vSphere. For this to happen, I first had to install and configure the vSphere CSI on Kubernetes using a Helm chart provided by Rancher. Although this is a valid configuration, the vSphere CSI has some disadvantages compared to making use of a CSI created by and for a specific storage solution like NetApp ONTAP which has snapshot and cloning capabilities.
Trident is NetApp’s open source storage orchestrator which integrates with Docker and Kubernetes, as well as platforms built on those technologies, such as Red Hat OpenShift, and Rancher. The goal of Trident is to make the provisioning, connection, and consumption of storage as transparent and friction less for containerized applications as possible. Trident automates storage management tasks for creation and provisioning against meta-classes of storage with common characteristics, providing abstracted pools of storage to the consumer. Users can take advantage of the underlying capabilities of NetApp storage infrastructure without having to know anything about it.
In this two part blog I’ll explain how install and configure Trident for Kubernetes, define a NetApp ONTAP NFS backend, create a persistent volume and resize it. In the second part, I configure Kubernetes for storage snapshots, make a snapshot of a persistent volume, clone it and provision it to a new workload as you would do in a CI/CD pipeline.
So, let’s get started!
Install Trident
Trident is an out-of-tree storage provider and there are two ways you can deploy Trident.
- Deploying Trident with
tridentctl
- Using the Trident operator
Using the Trident operator is the preferred way as the operator controls the installation of Trident, taking care to self-heal the install and manage changes as well as upgrades to the Trident installation. Starting with Trident 21.01, you can use helm
to install Trident operator.
To start you need the kubeconfig file of your Kubernetes cluster. When you have Rancher and deployed an RKE cluster then this is very easy.
Login to Rancher, select your cluster. Then click on Kubeconfig File and copy it to your clipboard. Merge it with your existing local config file under ~/.kube/
Change the context to your cluster. Tip: with [kubectx](https://github.com/ahmetb/kubectx)
you can switch very easy between multiple clusters!
Download the Trident installer.
wget https://github.com/NetApp/trident/releases/download/v21.01.2/trident-installer-21.01.2.tar.gz
tar -xf trident-installer-21.01.2.tar.gz
cd trident-installer
Create a new namespace for Trident and change the context to the trident namespace. Tip: with [kubens](https://github.com/ahmetb/kubectx)
you can switch very easy between multiple namespaces!
kubectl create namespace trident
kubens trident
In the trident-installer go to the helm directory. Install the Trident operator with helm
. Check if the trident operator and csi pods are running.
cd helm
helm install trident trident-operator-21.01.2.tgz
kubectl get pods
Use tridentctl
to check the installed version.
cd trident-installer
./tridentctl version
Create a NFS backend for Trident
With Trident installed it now needs to know where to create volumes. This information sits in objects called backends. A backend contains:
- The driver type
- Currently there are 12 different drivers available within Trident for the various NetApp portfolio of on-prem and Cloud based storage
- How to connect to the driver
- SVM, IP address, username, password etc.
- And some default parameters
- Auto export policy, aggregate/volume usage limits etc.
Starting with Trident 21.04 backends can be created and managed directly through the Kubernetes interface using command-line tools such as kubectl
. In my case I installed Trident 21.01.2 and have to create a backend using tridentctl
.
If you created a backend using tridentctl
and then upgrade to a newer version of trident (21.04+) the backend created with tridentctl
will continue to remain operational throughout the upgrade. Afterwards you can create a TridentBackendConfig
to manage pre-existing backends using kubectl
if you so desire.
First, login to NetApp ONTAP Sytem Manager and get the following details:
- SVM name
- SVM Management LIF IP
- SVM Data LIF IP for NFS
- SVM admin username and password
Create a json file for the backend. For example, backend-ontap-nas.json
{
"version": 1,
"storageDriverName": "ontap-nas",
"backendName": "ontap-nas-auto-export",
"managementLIF": "10.0.0.143",
"dataLIF": "10.0.0.144",
"svm": "svm1",
"username": "vsadmin",
"password": "Netapp1!",
"limitVolumeSize": "15Gi",
"autoExportCIDRs": ["10.0.0.0/24"],
"autoExportPolicy": true
}
In this example I use the Trident ontap-nas driver, connect to svm1 and limit the size of the a volume which can be created. I also enable autoExportPolicy
for dynamic export policy management.
For additional information, please refer to the official NetApp Trident documentation on Read the Docs:
Create the backend using tridentctl
.
./tridentctl -n trident create backend -f backend-ontap-nas.json
Create a Kubernetes Storage Class
For users to create Persistent Volume Claims (PVC) in Kubernetes, one or more Storage Classes are needed. A storage class points to a backend and contains the definition of what an app can expect in terms of storage, defined by some properties (access type, media, driver etc).
For additional information, please refer to:
Create a yaml file for the Storage Class. For example, sc-csi-ontap-nas.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ontap-nas-auto-export
provisioner: csi.trident.netapp.io
parameters:
backendType: "ontap-nas"
allowVolumeExpansion: true
Notice that I added the allowVolumeExpansion
parameter. Setting this on true makes it possible to resize volumes after creation.
Create the Storage Class using kubectl
kubectl create -f sc-csi-ontap-nas.yaml
kubectl get -n trident tridentbackends
kubectl get sc
Also in Rancher the Storage Class is visible.
Deploy an app with persistent file storage
For this task I will be deploying Ghost (a light weight web portal) utlilizing RWX (Read Write Many) file-based persistent storage over NFS in three steps:
- Create a Persistent Volume Claim (PVC) for blog content
- Create a deployment for Ghost using the blog content PVC
- Create a service to expose the Ghost application
Create a new directory for the Ghost app. Within this directory create three files:
- pvc.yaml
- deploy.yaml
- service.yaml
Edit the pvc.yaml file. Create a ghost namespace and the PVC using kubectl
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: blog-content
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: ontap-nas-auto-export
kubectl create namespace ghost
kubectl create -n ghost -f pvc.yaml
If we now login to ONTAP System Manager we should see a new Trident volume being created and automatically mounted, via the dynamic managed export policies, to the nodes of the Kubernetes cluster.
Edit the deploy.yaml file and deploy the Ghost app using kubectl
apiVersion: apps/v1
kind: Deployment
metadata:
name: blog
labels:
app: blog
spec:
replicas: 1
selector:
matchLabels:
app: blog
template:
metadata:
labels:
app: blog
spec:
containers:
- name: blog
image: ghost:2.6-alpine
imagePullPolicy: Always
ports:
- containerPort: 2368
env:
- name: url
value: http://my-blog.homelab.int
volumeMounts:
- mountPath: /var/lib/ghost/content
name: content
volumes:
- name: content
persistentVolumeClaim:
claimName: blog-content
kubectl create -n ghost -f deploy.yaml
Edit the service.yaml file and expose the Ghost app using kubectl
apiVersion: v1
kind: Service
metadata:
name: blog
spec:
#if you have a load balancer installed use type: LoadBalancer
type: NodePort
selector:
app: blog
ports:
- protocol: TCP
port: 80
targetPort: 2368
nodePort: 30080
kubectl create -n ghost -f service.yaml
Let’s go over to Rancher and see if the workload and it’s persistent volume are visible.
First, move the ghost namespace to a project. In my case the homelab project.
Select the cluster. In the top menu click Projects/Namespaces. Select ghost under Namespace: all and click on the submenu (3 dots) to move it to a project.
Select your project. In my case, homelab. Click Move.
Click on the project and select the Workloads tab. Under Namespace: ghost, the blog app is active.
Select the Volumes tab to show the persistent volumes (PV) and claims (PVC). Under Namespace: ghost, you should see the PVC blog-content and the related PV being bound.
This means the Ghost app is up-and-running. Let’s check and create some content.
Depending on if you have used the LoadBalancer
or NodePort
type in your service configuration, go to http://YOUR_EXTERNAL_IP/admin or http://YOUR_WORKERNODE_IP:30080/admin
Create a Ghost admin account by clicking Create your account.
Enter your account details and click Last step: Invite your team.
Skip this by clicking I’ll do this later, take me to my blog.
Create a new story and publish it.
Logout and point your web browser to http://YOUR_EXTERNAL_IP or http://YOUR_WORKERNODE_IP:30080
Confirm data persistence
To confirm data persistence we restart the blog pod on another node by draining the worker node where it’s running on.
First check if all cluster nodes are up and on which worker node the blog pod is running.
kubectl get nodes -o wide
kubectl get pod -l app=blog -n ghost -o wide
In my case the blog pod is running on node cls01-worker2. Let’s drain the node and set it in maintenance mode.
kubectl drain cls01-worker2 --ignore-daemonsets
All running pods, daemonsets excluded, are now being stopped and rescheduled on the remaining worker node(s).
Check if node cls01-worker2 is in maintenance mode and if the blog pod is now running on another worker node.
kubectl get nodes
kubectl get pod -l app=blog -n ghost -o wide
Refresh your web browser and access your blog post to confirm it still exists.
Resize the volume
Trident supports the resizing of File (NFS/RWX) & Block (iSCSI/RWO) PVC’s, depending on the Kubernetes version.
NFS resizing was introduced in Kubernetes 1.11, while iSCSI resizing was introduced in Kubernetes 1.16. Resizing a PVC is made available through the option allowVolumeExpansion
set in the Storage Class.
To resize a persistent volume you have to edit the PVC object.
kubectl get pvc -n ghost
kubectl edit pvc blog-content -n ghost
#under spec change the required size
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
When I configured the NFS backend in Trident, I set the limitVolumeSize
to 15Gi. So, what happens if I try to create a PVC that’s bigger than 15Gi?
Create a pvc_tobig.yaml file and try to create a new PVC using kubectl
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: big-blog-content
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
storageClassName: ontap-nas-auto-export
kubectl create -n ghost -f pvc_tobig.yaml
kubectl get pvc -n ghost
The new PVC object is in pending state.
Login to Rancher and investigate why the PVC in in pending state.
Select the cluster. In the top menu click on Storage. In the drop down menu select Persistent Volumes.
Click on the big-blog-content PVC in pending state and expand Events.
Under Events you see the Warning Provisioning Failed. In the Message you can see that the requested size was bigger than the size limit. Due to this the PVC will not be provisioned. Mission accomplished!
Next will be part 2 where I’ll explain how to enable storage snapshots & cloning in Kubernetes, make a clone of a persistent volume and use it for testing an upgrade of the Ghost blog app.