While working on and around monitoring Kubernetes clusters with Prometheus, I have noticed a reoccurring problem: metrics that are retrieved by Prometheus may potentially contain sensitive information (for example the Prometheus node-exporter exposes the Kernel version of the host), which a potential intruder may use in order to exploit their way through a respective Kubernetes cluster. So I asked myself the question: How does one authenticate and authorize requests from Prometheus so that Prometheus and only Prometheus can retrieve metrics from an application running in a Pod?

In this blog post, I will try to explain the relation between Prometheus, Heapster, as well as the Kubernetes metrics APIs and conclude with the recommended way how to autoscale workloads on Kubernetes.

The kube-state-metrics project is a service you can deploy in your Kubernetes cluster. It watches Kubernetes objects, in order to expose metrics in the Prometheus format. The project lives under the umbrella of the Kubernetes organization and was started by Sam Ghods at Box in May 2016. There has been a growing popularity in the project, so I wanted to share the current state of the project and what I believe the future holds. This post will particularly focus on the architectural changes that the project has undergone and explain the reasoning for the more significant ones.

Kubernetes supports multiple means of authentication, for example Static Token File, Static Password File as well as OIDC, which are all very well documented. But another common way of authentication is to make use of X509 client certificates. In this blog post I will explain a bit about X509 client certificates as well as demonstrate how to set them up for use in your Kubernetes cluster.