> There is just no substitute for understanding how everything works.<p>Great line, and one I strongly agree with. I love the tooling we have available today like Terraform, Ansible, etc. but the experience I gained by keeping bare metal servers alive and happy with nothing more than a shell has undoubtedly made me a much better admin.
This was a fascinating read but I feel like I’m left with some more questions about what’s going on under the hood in gitlab cloud:<p>1. I thought gitlab was hosted in google cloud, but there’s a lot of references to e.g., a hand rolled consensus system and self-managed database clusters. I’m wondering if this event changes the math on build vs. buy at all for gitlab, sounds like a lot of money has gone into this solution. How did that solution come up, is it about some specific Postgres features that aren’t available in googles hosted dbs or pricing?<p>2. Again on the google cloud angle, why are servers being hand-managed and rebooted? Elasticity in the cloud would make me think that the safest option would be to stand up parallel infrastructure (like in a DR plan) and migrate traffic. Was this just about speed of solution rollout? Does gitlab have plans to harden DR plans so that you can execute in cases like this? Whenever someone says they’re “in the cloud” and yet unable to treat servers like cattle, I get a bit worried.
> After looking everywhere, and asking everyone on the team, we got the definitive answer that the CA key we created a year ago for this self-signed certificate had been lost.<p>The GitLab outages always make the company seem disorganized and sloppy, and unable to reflect on how to improve how they work. So they don't have a central place to store their CA, and even after an outage, did they improve anything about how they work?<p>It's ironic that the post seems geared towards recruiting, though I guess it's honest, you know what you're getting into with that team.
> It is maintained by the Infrastructure group, which currently consists of 20 to 24 engineers (depending on how you count)<p>20 if you count in Base-12 and 24 if you count in Base-10?
It blows my mind they didn’t have sane PKI with that many resources. It seems like even the “small” initial team of a couple devs, a manager, <i>and a director</i> would’ve at least spun up a vault instance to use as a CA.<p>Also, easy for me say from the peanut gallery, but don’t understand why they couldn’t have done rolling consul restarts to update the configs, I’ve done this many times on consul clusters.
Consul issues have bitten me at two companies, and I heard word of it being the culprit for some serious outages elsewhere. One possible takeaway here is to remove it.
If they were going through all this trouble and worry why not create a new CA and drop the crrts from it on the hosts? That’s the work of just a few minutes (plus some bash scripting to mass generate your host certs). If they had already accepted that they were going to restart the services on all the hosts anyway it would have saved them having to restart all the services again in the future when they need to drop more certs.
Maybe I am saying something stupid, but infrastructure services should be able to use a dynamic set of keys.<p>If the first doesn't work, you try the second, and then the third and so so.<p>Similarly the clients, we should be able to dynamically adds certificate.<p>Our own key expires and our services are all about to drop connection seems something that should not happen.