They hopped call with AWS Suport folks about their use of S3 and yet this happened:<p>“ 17:00: While rewriting the migration we find other issues since S3 allows multiple concurrent writes to the same project. This is a race condition that would have been prevented by Github since it handles out of sync commits thanks to it being an actual git repository. On Github, if a pushed head already exists as a commit it won’t overwrite a newer head, but there are no controls like this in S3. The impact mostly seems to be for new projects.”<p>That is a relatively well known failure pattern with git on s3 and it’s a shame the s3 representatives weren’t aware or there isn’t a knowledge base to tap into. I have found with AWS you have to get beyond the first level of support to have any chance of having a meaningful conversation.<p>When it comes to user data, you have to be extremely careful, so I was surprised on X how quickly they were moving things off of GitHub in a relatively celebratory manner. They had a ton of attention from this incident so I guess they wanted to recover as fast as possible versus being sure they couldn’t lose user data (and to be fair it seems like it was all recoverable). Still should be careful as their service becomes more popular and important.