Gain insight in Kubernetes

In my previous post I showed how easy it is to deploy and manage Kubernetes clusters and namespaces using VMware vRealize Automation. In my case, Kubernetes is automatically installed on Virtual Machines running on a VMware vSphere infrastructure. These Virtual Machines run pods, containers, storage- and network services used by applications running inside these containers and connecting them to the outside world for end-users to consume. As you can see there are a lot of pieces and bits that make up a Kubernetes application environment and a lot can go wrong. For monitoring and troubleshooting it’s important to gain insight in all these different layers. But how? 

In this blogpost I’ll show you some tools which you can use to gain more insight in Kubernetes.

Kubernetes Dashboard

Dashboard is a general purpose, web-based user interface for Kubernetes clusters. It allows users to deploy and manage containerized applications running in the cluster and troubleshoot them, as well as manage the cluster and its resources itself.

If you used vRealize Automation to deploy my Kubernetes blueprints, then Dashboard is already installed and a bearer token is automatically generated to get access to Dashboard.

There are multiple ways to access Dashboard. The first option is to spin up a proxy server between your local machine and the Kubernetes API Server using the kubectl proxy command.

Gain insight

The second option is to edit the service that’s was configured during the installation of Dashboard and change it to use a NodePort or LoadBalancer. The NodePort option creates a static port forward to the service. To access Dashboard use the IP address of one of your Kubernetes nodes :.

The LoadBalancer option exposes the service externally using a software load balancer or a cloud provider’s load balancer solution like VMware NSX or AWS ALB. It automatically provisions a load balancer for the service and maps it to an assigned NodePort :. It balances traffic across all pods that are mapped to the service. 

In my Kubernetes blueprints I install MetalLB as software load balancer to use.

  1. Login to the Kubernetes master node using SSH.
  2. Edit the Kubernetes Dashboard service using the command:
    • kubectl edit service kubernetes-dashboard -n kubernetes-dashboard
  3. Change it to use LoadBalancer and save it.Gain insight
  4. View the IP address and NodePort mapped to the service using the command:
    • kubectl get services -n kubernetes-dashboardGain insight
  5. Access Dashboard (https://<Your_Load_Balancer_IP>) and use the bearer token available on the master node.Gain insightGain insight

VMware Octant

VMware Octant is an alternative for Kubernetes Dashboard. Octant is a tool for developers to understand how applications run on a Kubernetes cluster. Octant offers a combination of introspective tooling, cluster navigation, and object management along with a plugin system to further extend its capabilities.

  1. To use Octant, first install it on your local machine.
  2. Get the KUBECONFIG file of your Kubernetes cluster. If you registered your cluster in vRealize Automation, select the Infrastructure tab in Cloud Assembly, click on Kubernetes, select your cluster en click on the KUBECONFIG link to download.Gain insight
  3. Start Octant on the CLI using your KUBECONFIG file.Gain insight
  4. Your local webbrowser wil automatically be opened with Octant.Gain insightGain insight

VMware vRealize Operations

Another tool you can use to gain insight is VMware vRealize Operations. With VMware vRealize Operations Management Pack for Container Monitoring, cloud admins can get complete Kubernetes topology of Namespaces, Clusters, Replica Sets, Nodes, Pods, and Containers for monitoring Kubernetes clusters. The out of the box dashboard not only provides an overview of Kubernetes ecosystem but also troubleshoots by highlighting the Key Performance Index and alerts for various objects pertaining to Kubernetes clusters that are monitored. This management pack extends the monitoring capability of vRealize Operations to provide insights in the Kubernetes clusters to the Virtual Infrastructure administrator.

Before installing and configuring the vRealize Operations Management Pack for Container Monitoring you must install cAdvisor as DaemonSet on your Kubernetes cluster.

  1. Copy the YAML code for the cAdvisor DaemonSet here
  2. Log on to the master node using SSH, create a YAML file for the DaemonSet and run it using the following command:
    • kubectl apply -f <Your_File_Name.yaml>
  3. Check if the cAdvisor pods are running on each node.Gain insight

If the DaemonSet was successfully deployed, install the management pack.

  1. Download the vRealize Operations Management Pack for Container Monitoring from VMware Solution Exchange
  2. In vRealize Operations, select the Administration tab and in the left menu under Solutions select Repository. Scroll down and click Add/Upgrade. Browse to the PAK file to install the management pack.Gain insight
  3. Under Solutions, select Other Accounts and click on Kubernetes Adapter.Gain insight
  4. Enter a name for your Kubernetes account. Enter the URL of your master node, select DeamonSet as cAdvisor Service and enter the port as defined in the cAdvisor DeamonSet YAML.Gain insight
  5. Click on the + to add a credential. Use the token, which was automatically generated, available on the master node.
  6. Enter the FQDN of your vCenter Server.
  7. Validate and accept the certificate. Then Save.
  8. To open the out of the box dashboard, go to Dashboards and select Kubernetes Overview.
    Gain insightGain insightGain insight

VMware vRealize Log Insight

Kubernetes is a highly distributed and dynamic environment. In production, you’ll most likely be running dozens of machines with hundreds of containers that can be terminated, restarted, or rescheduled at any point in time. This transient and dynamic nature of the system is a challenge in itself. As with any system, logs help cloud admins and developers to gain observability and insight into containers and the Kubernetes clusters they’re running on. 

But how do you gather Kubernetes related logs and send them to a log aggregation tool like VMware log Insight? The answer is Fluentd

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. Once installed on a server, it runs in the background to collect, parse, transform, analyze and store various types of data.

There’s only one problem. VMware Log Insight doesn’t understand Fluentd output out of the box. Luckily there’s a Fluentd plugin available for Log Insight. To use this plugin we first have to install it somewhere so it can be used by Fluentd and tell Fluentd, using a configuration file, what data to gather, analyse and transform it, and send it over to Log Insight. For this I use a container build by a Dockerfile which then can be used in a DaemonSet to run on my Kubernetes cluster nodes. To gather audit data I also have to create a Kubernetes Audit policy and apply it to the cluster.

  1. Log on to the master node using SSH.
  2. Go to the etc/kubernetes directory and create a new directory for the audit policy named policies: mkdir /etc/kubernetes/policies
  3. Copy the Audit policy YAML and create a new file, audit-policy.yaml, in the policies directory: vi /etc/kubernetes/policies/audit-policy.yaml
  4. Go to the etc/kubernetes/manifests directory and edit the kube-apiserver.yaml file.
  5. Under command: add the following lines:
  6. Under volumeMounts: add the following lines:
  7. Under volumes: add the following lines:
  8. Quit and save.
  9. Restart the kubelet service on the master node to apply the changes: systemctl restart kubelet

You may need to wait a bit before the audit log file will be available. But if all was done right, you can find your audit log files in the /var/log/apiserver directory.

Now we’re ready to create the Fluentd daemonSet.

  1. Copy the YAML of the Fluentd DaemonSet here.
  2. Log on to the master node using SSH, create a YAML file for the DaemonSet. 
  3. Before running it, make sure you added your VMware Log Insight server IP address or FQDN. Quit and save.
  4. Run it using the following command:
    • kubectl apply -f <Your_File_Name.yaml>
  5. Check if the Fluentd collector pods are running on each node.
  6. Log in to Log Insight and check if logs are being collected from the Kubernetes nodes. Go to Administration, in the left menu select Hosts. Scroll all the way down and view the details of hosts sending events without a hostname field.
  7. Select the Ip address of one of your Kubernetes nodes, for example the master node and view the logs in the Interactive Analytics tab. For example query for deployments named nginx recently scaled up.Gain insight
  8. From these types of queries you can build your own dashboards in Log Insight.Gain insight

As you have seen the vRealize Cloud Management solutions are very well equipped to help you gain insight in Kubernetes. From infrastructure up to container applications and all things in between. If you want more insight from an application perspective, Tanzu Observability by Wavefront is probably more suited.