So you wanna write Kubernetes controllers?

At work we are using Metacontroller to implement our &quot;operators&quot;. Quoted because these are not real operators but rather Metacontroller plugins, written in Python. All the watch and update logic - plus the resource caching - is outsourced to Metacontroller (which is written in Go). We define - via its CompositeController or DecoratorController CRDs - what kind of resources it should watch and which web service it should call into when it detects a change. The web service speaks plain HTTP (or HTTPS if you want).<p>In case of a CompositeController, the web service gets the created&#x2F;updated&#x2F;deleted parent resource and any already existing child resources (initially none). The web service then analyzes the parent and existing children, then responds with the list of child resources whose existence and state Metacontroller should ensure in the cluster. If something is left out from the response compared to a previous response, it is deleted.<p>Things we implemented using this pattern:<p>- Project: declarative description of a company project, child resources include a namespace, service account, IAM role, SMB&#x2F;S3&#x2F;FSX PVs and PVCs generated for project volumes (defined under spec.volumes in the Project CR), ingresses for a set of standard apps<p>- Job: high-level description of a DAG of containers, the web service works as a compiler which translates this high-level description into an Argo Workflow (this will be the child)<p>- Container: defines a dev container, expands into a pod running an sshd and a Contour HTTPProxy (TCP proxy) which forwards TLS-wrapped SSH traffic to the sshd service<p>- KeycloakClient: here the web service is not pure - it talks to the Keycloak Admin REST API and creates&#x2F;updates a client in Keycloak whose parameters are given by the CRD spec<p>So far this works pretty well and makes writing controllers a breeze - at least compared to the standard kubebuilder approach.<p><a href="https:&#x2F;&#x2F;metacontroller.github.io&#x2F;metacontroller&#x2F;intro.html" rel="nofollow">https:&#x2F;&#x2F;metacontroller.github.io&#x2F;metacontroller&#x2F;intro.html</a>

I used to be fascinated by the automation power of Kubernetes custom components. The declarative approach and reconciliation loop offers so many possibilities for creating higher level descriptions of domain specific infrastructure.<p>On reflection though, I think this stuff can lead to a lot of complexity layers which don&#x27;t benefit the product relative to the time investment. You are probably not Google.

Controllers are a complex topic, but as the linked talk describes, it all comes down to some basic control theory concepts. I wrote about them in my Desired state systems post <a href="https:&#x2F;&#x2F;branislavjenco.github.io&#x2F;desired-state-systems&#x2F;" rel="nofollow">https:&#x2F;&#x2F;branislavjenco.github.io&#x2F;desired-state-systems&#x2F;</a> if somebody wants a high-level overview of how to think about them.<p>Basically, declarative state implies value semantics which makes it easier to reason about. Underlying complexity is high though, and you need to judge how necessary it is.

I&#x27;d please ask people to don&#x27;t write operators unless absolutely necessary.<p>I used a certain tool which had its own config format, and it&#x27;s &quot;cloudnative&quot; operator implemented CRDs of which multiple can exist and they would update the config file in some mounted volume. Such thing is a hell to debug. Why can&#x27;t we just store the config file in configmap&#x2F; secret and listen to changes?<p>(If we had a better templating solution than helm, I think quite a few operators wouldn&#x27;t need to exist.)

No not really

Why do devops keep piling abstractions on top of abstractions?<p>There&#x27;s the machine. Then the VM. Then the container. Then the orchestrator. Then the controller. And it&#x27;s all so complex that you need even more tools to generate the configuration files for the former tools.<p>I don&#x27;t want to write a Kubernetes controller. I don&#x27;t even know why it should exist.

Low barrier to entry was not a phrase I was expecting in that article.<p>Either way I’m going to try my hardest to avoid this. K8s is hard enough to get right as is