Using kube-rbac-proxy to secure Kubernetes workloads

While working on and around monitoring Kubernetes clusters with Prometheus, I have noticed a reoccurring problem: metrics that are retrieved by Prometheus may potentially contain sensitive information (for example the Prometheus node-exporter exposes the Kernel version of the host), which a potential intruder may use in order to exploit their way through a respective Kubernetes cluster. So I asked myself the question: How does one authenticate and authorize requests from Prometheus so that Prometheus and only Prometheus can retrieve metrics from an application running in a Pod?

The default answer for authenticating and authorizing metric endpoints in Prometheus is using TLS client certificates, however, as issuing, verifying and rotating client certificates can get rather complicated, as a result, Prometheus requests are most of the time simply not authenticated and unauthorized.

I built the kube-rbac-proxy, a small HTTP proxy for a single upstream, that can perform RBAC authorization against the Kubernetes API using SubjectAccessReviews. In this post I want to explain, how it uses Kubernetes RBAC to accomplish this.

How does RBAC work behind the scenes?

Kubernetes role based access control (RBAC) itself solves only half of the problem. As the name suggests, it is only about access control, meaning authorization, not authentication. Before a request can be authorized, it needs to be authenticated. Simply said: we need to find out who is performing this request. The mechanism for services to authenticate themselves in Kubernetes are ServiceAccount tokens.

The Kubernetes API exposes the ability to verify ServiceAccount tokens, using what is called a TokenReview. The response of a TokenReview is simply whether the ServiceAccount token was successfully verified, as well with as which user the specified token is associated. The kube-rbac-proxy, expects the ServiceAccount token to be specified in the Authorization HTTP header, and then verifies it using a TokenReview.

At this point, a request has been authenticated, but not yet authorized. Parallel to the TokenReview, Kuberenetes has a SubjectAccessReview, which is part of the authorization API. In a SubjectAccessReview an intended action is specified as well as the user who wants to perform it. In the concrete case of metrics being requested by Prometheus, the /metrics HTTP endpoint is requested. Unfortunately that is not a fully specified resource in Kubernetes, however, the SubjectAccessReview resource is also able to authorize so called “non-resource-urls”. The possibilities are endless though, for example: authorizing a proxied request a SubjecAccessReview could also check to ensure a user has the services/proxy RBAC role.

When monitoring Kubernetes with Prometheus, then the Prometheus server likely already has permission to access the /metrics non-resource-url, as retrieving metrics from the Kubernetes apiserver requires the same RBAC role.

kube-rbac-proxy in action

Now that all necessary pieces have been explained, let’s see how the kube-rbac-proxy concretely authenticates and authorizes a request, with the case stated at the beginning of this blog post: Prometheus requesting metrics from the node-exporter.

When Prometheus performs a request against the node-exporter, with the kube-rbac-proxy in front of it, the kube-rbac-proxy performs a TokenReview with the provided ServiceAccount token, and if the TokenReview is successful, it continues to use the SubjectAccessReview to verify, that the ServiceAccount is authorized to access the /metrics HTTP endpoint.

Visualized, the full flow of authenticating and authorizing requests from Prometheus then looks like this:

Conclusion

If what you want to do is authorize certain requests with a static authorization profile, you may want to check out the kube-rbac-proxy.

While it may still be useful to perform authorization within your workload - especially when more fine grained or data specific access is required - it is good to know that in Kubernetes you have tools available that can get you a long way, before having to spend time on it.

Thanks for reading! If there are any question or feedback, feel free to let me know on Twitter @fredbrancz.

Further reading

• More in depth documentation on the specifics of the SubjectAccessReview can be found in the admin documentation
• The code for the TokenReview type
• The code for the SubjectAccessReview type