This post certainly rings a bell! Even with deep Kubernetes experience, we struggled at times with Istio at $previous_job, especially around:<p>- Control-plane performance on a cluster with a large-ish number of pods (thousand+) can be hit-and-miss, and "what to scale up" was hard to pinpoint (though admittedly it seems to be getting better)<p>- Istio upgrades often are a pain, but mostly around the actual way of deploying the upgrade, rather than the upgrade itself. For a long time there was no official Helm chart, then there was a Helm chart, then two Helm charts, now it looks like Helm is deprecated and will be removed; instead installing via `istioctl` is recommended... Some of it is due to the pain of upgrading CRDs, which is a general Kubernetes issue, but there's still a _lot_ of churn to keep up with.<p>- Adding a new VirtualService registered to the same hostname as an existing one will be accepted by Istio (at least as of 1.4), and will proceed to _silently break all routing for new pods joining your istio cluster_! This was a bear to debug too, given how noisy and confusing the Pilot logs are, and we ended up stitching up a custom Prometheus alert around this given it bit us roughly every other week<p>- HA for control-plane components isn't explained in the docs. Is it safe to run two Citadel pods? We did it and it seemed fine, but who knows?<p>- We sometimes ran into pathological cases where traffic would for some reason completely drop after a new deploy, and gradually pick up after the config was stremed onto sidecars, over a span of ~10-15 minutes. We never managed to debug this issue (which happened probably half a dozen times over a year), and that mere fact turned me off the complexity of Istio in general.<p>(Some of these might have been fixed in Istio 1.5+, as 1.4 was my latest experience)<p>Of course, once your setup is stable, everything is awesome: sidecar injection works flawlessy, observability is awesome, distributed tracing is a breeze, Kiali is a great crowd-pleaser when showing off features, mTLS + TLS origination mean full on-the-wire encryption without losing any of the previous benefits... A lot of features that meant we carried on with it, but if I had to start again I'd probably have a good hard look at Linkerd before recommending Istio for any prod setup.