This is very familiar. Just last week, we had the following:<p>Most of our test pipeline was broken, due to a long series of weird events, in which the uninstall phase of the previous version of one of our apps, on our CI computer was stuck, and on another CI computer somehow the CI user gitconfig file was somehow corrupted and this failed another test which tested our git interface. We're not sure why all this happened. Also we have started using nugets (I don't like them but the guy who's in charge of the CI wants to use them). And something was completely broken with the permissions on the internal nuget server we use. We're still not sure what. But IT took 2 weeks to sort it out.<p>Our team is very small. It's only one guy who deals with the CI and he does it part time. I used to do it, and then we had another excellent new guy who took it upon itself and really built amazing and mostly stable pipelines for all our products all by himself. But he left several months ago and management didn't hire a replacement. So now our CI pipeline is sometimes broken for days without anyone noticing.
It feels like every day something new breaks in our pipeline. We have weeks of stability and then months of tests failing for random reasons (network latency, concurrency, etc.)<p>I want to understand how universal these pain points are VS unique to the team I am working with at the moment.