How to Collect Kubernetes Node Metrics with Node-Exporter Using CronJobs

How to Collect Kubernetes Node Metrics with Node-Exporter Using CronJobs This article focuses on leveraging cron-jobs in Kubernetes to pull host metrics from all nodes in Kubernetes and save them as log files. A few days ago I was asked to solve this problem as part of an interview process for an SRE position. The problem is a very nice way to brush up your skills in monitoring. So, I decided to share it with all readers. TL;DR If you want a minified version of this solution then you can head over to this Github Repository Here it also includes all the code shown in this article. Audience This article is intended for Intermediate Kubernetes users with basic knowledge of Kubernetes Architecture, Kubernetes Networking, Containerization and Basic Linux Administration. The article assumes that the readers can access remote VMs using SSH. Also, some knowledge of how applications are developed in general will aid in understanding a lot of terminology used here. Also, I assume the reader understands the meaning of Push-based and Pull-based metrics collection architectures. Prior experience (basic) using Observability/Logging/Monitoring Stacks such as ELK, EFK or Prometheus-Loki-Grafana is required. Problem statement Create a Kubernetes cron job that pulls node metrics like (CPU, Memory, Disk usage) and stores them in a file. Expected Output: Note : Prerequisites I am using a K3d cluster which is a very lightweight Kubernetes distribution based on Rancher K3s that runs a K8s cluster on Docker. The following article uses a K3D cluster but this solution can be implemented using any Kubernetes Distribution with minimal configuration changes. I have the following configuration: - The Solution Architecture To tackle this situation we need an agent that would run on all nodes and continuously gather metrics. So, I decided to go with node-exporter designed to expose the metrics from a Linux Host. Since the targets here are Kubernetes Nodes we need to deploy node-exporter Pods as DaemonSets allowing us to run a node-exporter Pod on all nodes. Now to expose endpoints I created a Service to make endpoints easier to scrape from our Cron-Job Pods. [Optional] I also added a ServiceMonitor using the prometheus operator for Kubernetes so that this solution could be later integrated with the same. But, it is not needed here. The Cron Job itself is a simple Python script that scrapes the metrics from all node-exporter Pods one at a time by calling the /metrics endpoint. To save the metrics as logs I attached a PVC (Persistent Volume Claim) to each pod. Also, we will push these logs to an AWS S3 Bucket as a backup. Step 1: [Optional] Install Prometheus Operator This step is only needed if you want to use this configuration alongside Prometheus later without the hassle of configuring things yourselves. File:-operator-values.yml This is a HELM Values Manifest that ensures that only the CRDs are installed This will install the following CRDs on your cluster Step 2: Configure S3 Bucket storing these logs. We are using S3 Buckets to push logs. This is meant to serve as a backup for storing the logs in the event of data corruption on the node holding the Persistent Volume (PV). Also, configure an AWS IAM User to allow access to the S3 bucket via the Python Boto3 Library. Also, you should configure Access Keys for accessing your AWS IAM User and grant sufficient permissions for your user to access the logs. Step 3: Writing the CronJob Code I chose Python as the Kubernetes API integrates very well without the hassle of configuring kubectl inside the container. Head to this directory in linked repository code here.Here I have also included tests for the same code which you can run. Dockerfile for the Image Let's Dockerize the code. We will make sure we do not use the root user for running the scripts inside the container. Step 4: Deploy Node Exporter Step 4.1: Create a DaemonSet A DaemonSet is a Kubernetes Controller that runs a Pod on all (or some) nodes in the Kubernetes Cluster. On Applying you will see the following Pods in your cluster Step 4.2: Create a Service Why create a service in the first place? Can't you access Pods directly by assigning endpoints or better access them using their IP Address inside the cluster? And you might be partially correct. All you need is endpoints to connect to the pods in the DaemonSet. The answer is short Services make it easy to manage endpoints in the Kubernetes Cluster. Here we will create a service of type ClusterIP for the pods. You may want to check the endpoints via this command The CronJob does not need to access the Service. Rather, it has to scrape logs from each Pod that runs node-exporter this is why we need to directly query the endpoints instead of the Service. Each Pod exposes a /metrics API endpoint. Which can be tested as follows let us call one of the endpoints:- Here's the output:- Step 4.3: [Optional] Create a ServiceMonitor This step is only needed if you have installed the Prometheus Operator as discussed above and later want to deploy Prometheus. Step 5: Let's Deploy Our CronJob Step 5.1: Create a Service Account for our CronJob Pods We will create a Service Account, a Role and a RoleBinding to allow read-only access only to endpoint objects in the namespace where the node-exporter Pods are Service Account Role Role Binding Step 5.2: Configure Secrets and Configurations This way of configuring Secrets is not very secure consider using a Secrets Manager in Production Secrets ConfigMap Step 5.3: Create the Persistent Volume and Persistent Volume Claims We will be utilising Persistent Volumes and Persistent Volume Claims.They will be used to store our logs on nodes. Here I am using the rancher.io/local-path storage class on K3D. Change this configuration according to the CSI driver that you are using on your cluster. Step 5.4: Create the CronJob Now the CronJob will schedule a pod every 5 min which is configured via the cron expression */5 * * * * Verify the deployment of CronJob Wait, for the Pods to come up as scheduled by the CronJob we will find that the pods exit after a successful execution. Accessing the PVs containing the Logs Now, we need to access these logs on our nodes. I am using K3d so my nodes are docker containers but on production-level clusters, these nodes are generally VMs on which you may need to SSH. Determining the node where our Persistent Volume is provisioned Execute the following commands to get the name of the Node where the PV is deployed Step 0: Get the name of PV Step 1: Get the Node and the Mount Point of the Volume on the Node. We see in this case PV is mounted inside the control plane node k3d-multicluster-server-0 at the directory /var/lib/rancher/k3s/storage/pvc-9f6288f3-75d2-4c4b-b39a-28d823ac2252_node-exporter_node-exporter-cron-pvc Step 3: SSH into the node (in my case k3d node was a docker container so I had to do a docker exec). Step 4: Let's access the logs Output: Accessing the Logs inside the S3 Console Issues with this architecture Although this architecture has an S3 Backup configured, it still faces Single Point of Failure problems that need to be addressed. This means if data on a node is destroyed the logs for all nodes are lost which defeats the very purpose of Logging and Metrics Collection. This can be scaled using the following approaches: -

How to Collect Kubernetes Node Metrics with Node-Exporter Using CronJobs

Topics included

Social.
Snapshot.

Explore similar content

How to Collect Kubernetes Node Metrics with Node-Exporter Using CronJobs

Topics included

Social.Snapshot.

Explore similar content

Social.
Snapshot.