For anyone managing a k8s cluster and are fatigued with memorizing and reciting kubectl commands should definitely take a look at k9s[0]. It provides a curses like interface for managing k8s which makes it really easy to operate and dive into issues when debugging. Move from grabbing logs for a pod to being at a terminal on the container and then back out to looking at or editing the yaml for the resource definition in only a few key presses.<p>[0] <a href="https://k9scli.io/" rel="nofollow">https://k9scli.io/</a>
Lots of people complain about Kubernetes complexity but I have found it is as complex as you make it. If you are running some simple workloads then once you have the pipeline setup, there is almost no maintenance required.<p>When people complain and then start talking about super complex configuration, bespoke networking functionality and helm charts that have "too many options" then surely that just means you don't have the skills to use the system to that degree?<p>I could say that .Net is too complicated because it has MSIL and library binding sequences involving versions and public keys and the fact you can not always link e.g. netfx with netstandard but these are either just things you need to learn, or things that you can't use until you do learn them.<p>It's like someone complaining that a Ferrari is too complicated because you can't work out how to change the cylinder head when most people will just drive it.
...or deploy your code on Google App Engine, Heroku, Elastic Beanstalk, Digital Ocean App Platform, Fly.io (etc etc) and spend all your time implementing user-facing features instead of maintaining infrastructure.<p>Yeah, I get it, compared to maintaining bare metal, k8s is amazing. But you're still wasting your time working on plumbing.
Amazon EKS + Fargate.<p>No bare metal to manage.
Control plane complexity is abrastacted away.
Fargate namespace + profiles, no worker node configuration.<p>EKS cost will be $90/m
Thereafter you only pay for what cpu/mem limits you assign to your deployments/pods.<p>Otherwise, why bare metal? For your basic needs, bare metal, self managed control planes, etc are definitely over complicating things.
> This deployment needed to serve docker images on boot-up, which instance templates do support. However they don't support them via API, only in the web console or the CLI.<p>Not exactly true. I got around this limitation by sending a startup script[1] in API metadata which basically just does invokes `docker run ...` and it works just fine. This allows spinning up/down container-based VMs via API only which is nice.<p>[1] <a href="https://cloud.google.com/compute/docs/instances/startup-scripts/linux" rel="nofollow">https://cloud.google.com/compute/docs/instances/startup-scri...</a>
> <i>Keep it managed since that legitimately becomes a reliability headache.</i><p>This is the thing that I think will always give me pause. If I have to pay a third party to manage my cluster orchestration backplane, that seems like a pretty big piece of overhead.<p>Sure, I can do it myself, but then I have to deal with said reliability headache. It seems ironic that a cluster management framework -- that touts its ability to reliably keep your applications running -- has its own reliability issues.
Vanilla k8s is pretty good. But once the 8 trillion vendors have you 'curl | helm' ing you end up with a knot of a system.<p>Keep it simple, use GitOps (ArgoCD is great), let k8s do what it's good at, managing workloads, not as a delivery mechanism for a vendor.<p>As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template.
We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I have to say it is a love-and-hate story.<p>On the bright side, k8s is almost the only option of an abstraction layer on top of different Clouds, for a complex system with tens of components. Database is more than masters and workers, there are so many components you need to take care of. For example, we may need monitoring agents, certificate managers, health checkers, admin proxies, etc. Without k8s, you have to be the owner of a kindergarten.<p>On the other side, k8s IS complicated. It's like an unexplored garden. People just enter it and try to use whatever they see, and cause all kinds of problems. What we met are:<p>* Try to apply the operator pattern to everything, debugging is really painful. Learning curve is steep.
* Small services still cost a lot. VPA is not mature enough and many tiny services may be just better off on lambda.
* k8s is not really designed for small per-tenant clusters. Managing a fleet of clusters is no easy job, but it is something SaaS companies have to deal with.
From the Google trend[0] and the historical data of Kubernetes repo on GitHub[1], K8s has crossed the chasm to become the dominant choice for infrastructure. Whether it is the result of several companies working together, or from developers' choice. I think K8s will remain mainstream until there are big changes in the infrastructure world.<p>[0] <a href="https://trends.google.com/trends/explore?date=2014-06-01%202022-08-10&q=Kubernetes" rel="nofollow">https://trends.google.com/trends/explore?date=2014-06-01%202...</a>
[1] <a href="https://ossinsight.io/analyze/kubernetes/kubernetes" rel="nofollow">https://ossinsight.io/analyze/kubernetes/kubernetes</a>
Seems like everyone is forgetting about PaaS, and I don't understand why ..<p>For many use-cases it's going to be much simpler and cheaper than a manged k8.<p>There's no lock-in with Cloud Run than GKE. (actual lock-in comes with proprietary databases and the like.)<p><i>edit</i>: Missed the GPU part, might make the OP's project the exception to the rule<p>People <i>also</i> forget about auto-scaling groups of VM's such as Managed Instance Groups in GCP: <a href="https://cloud.google.com/compute/docs/instance-groups/" rel="nofollow">https://cloud.google.com/compute/docs/instance-groups/</a>
"Azure is the only major provider that still has free control panels"<p>Oracle Cloud Infrastructure does as well. Perhaps it does not yet qualify as major... It's major to Oracle, that's for sure.
This is one of the pet peeves of HN submitters and readers :)<p>Sure, here is my two cents FWIW - Kubernetes is complex for some set of folks but not for others. So, the answer is - <i>depends</i>; On a lot of external factors outside of just technical capabilities of team.<p>Kubernetes solves many non-trivial challenges but its not a silver bullet. I could certainly learn from Mercedes platform/infra team's "story from the trenches" (they reportedly run 900+ k8s clusters on production:)
The fact that there is a think piece everyday either extolling or cursing Kubernetes is a key indicator this bit of tech has some serious teething issues.
i still believe most of the companies are better off with just deploying containers on vm's with autoscaling groups with a LB in front of it and some kind of terraform + ansible | CI-pipeline deployment to manage it.<p>all the complexity you have to buy with dealing with k8s.
you have to update it frequently, update all the charts, fix all the breaking things in a developing ecosystem, and still have to deal with the hard parts of dynamic certs, storage and network. and you would still need to dedicate "normal" vms to host databases and your monitoring and your storage because if your cluster goes down, so is your management layer.<p>i have been with k8s since 1.3 and it is so disproportionate that i will not touch it for most of the times
For very bare minimum playing around, I use k3s on Linode (1 shared CPU, 1 GB RAM, 25 GB disk for a master and another one for worker node) at 10$ / month.
> Bare metal is always my first choice both for past projects and startups.<p>Is this a new Tech Hipster thing? Like writing a letter with a typewriter rather than an ink pen or computer/printer? "You don't understand, man; a virtual machine is, like, <i>not authentic</i>, man."
My problem with Kubernetes is my problem with with front-end web frameworks - they introduce too much complexity to the point of being esoteric for simple systems.<p>If you have a simple website built on boring technologies like HTML, CSS, and vanilla JS, then nearly anyone can read, understand, and make changes to it, even backend developers. If you instead wrote it in React/Webpack/etc. then suddenly only frontend experts can understand and contribute and debug.<p>Same with k8s. If you make a cloud backend using boring technologies like plain old programs calling APIs then nearly anyone can read, understand, and make changes to it. But if you instead make it use a big pile of configuration files with a million switches then suddenly only k8s experts can understand and contribute and debug.<p>I'm not saying don't use ReactJS or Kubernetes, I'm just saying make sure that the benefits you get from switching to it outweigh the new complexity and therefore expertise required to understand and debug it.