Why not Kubernetes?

6th of February, 2022

First, I would like to say I think Kubernetes is an excellent platform for its intended purposes. It provides excellent fault-tolerance all over the cluster, a fast and easy way to run updates on your deployments, great tools for managing services, volumes, metrics, and more, each having its own lifecycle to manage. Also, implementing your tooling by extending the Kubernetes' API is a trivial task, so you can easily leverage the great tooling to make your own for whatever you might need.

Today it's also effortless to spin up a Kubernetes cluster with various installers and different managed options. While being very complex, it's still a step closer to the idea of "just run my code and make it work". Also, with containers in the picture, we are pretty close to the magical situation where we actually can run the same application similarly on the laptop and in one cloud.

For me, issues start rising when we begin using Kubernetes for something other than its intended purposes. While I don't have any statistics on this, I have a pretty strong gut feeling that most of the people running Kubernetes are using it as a glorified scheduler for placing containers on nodes as fast as possible. While it's an excellent and overall pretty easy tool to use for orchestrating containers, its fundamental purpose is to orchestrate anything crucial to your infrastructure like network, storage, and other dependencies.

Kubernetes allows complete user freedom to run your infrastructure as you see fit. Despite sounding like a cliche, this kind of freedom can bear huge responsibilities. I would dare to say that most developers and system administrators don't want to make these decisions. What if, at some point in the development, you would wish to change your networking interface or maybe some dynamic storage provider? Can you even do such a thing in that stage of development if the decision was made before you even had anything running in Kubernetes?

Kelsey Hightower put it nicely a while back when he described Kubernetes isn't meant for being a developer platform but a framework for creating platforms. So while it definitely can work as a developer platform, and overall it's pretty easy to get started, kubectl run and kubectl expose and your good to go. That being said, all the API designs in Kubernetes are created for clusters and how to manage these. So while containers are part of this, there are so much more to be leveraged. So should application developers, startups or small businesses use something like this? Probably not, unless they are developing a platform product.

When we get into cluster management, we need to start thinking about managing the lifecycle of everything running inside the cluster. Unfortunately, this is also when things start to get hard. What to do if something inside the cluster dies? What if I need to provision something dynamically? Kubernetes is pretty good at simplifying many of these topics, but due to the complexity of things happening behind the scenes, all this complexity cannot be simplified away.

Kubernetes has a high entry threshold, and it's a very complex project, but still, way too often, I see it marketed as a simple solution for many problems. While you can use Kubernetes in a very simple manner and get lots of stuff done, eventually, you will hit a wall. Deploying fault-tolerant distributed applications that are scalable against a pool of machines with dependencies in networking, storage, and more, that's a hard problem.

Kubernetes is built for production workloads and running infrastructure beyond your demo application. For this reason, complexity in Kubernetes is justified and should be approached with that mindset.

If you have any questions or suggestions, write to topi at topikettunen dot com.

Tags: computers