They hopped call with AWS Suport folks about their use of S3 and yet this happened:<p>“ 17:00: While rewriting the migration we find other issues since S3 allows multiple concurrent writes to the same project. This is a race condition that would have been prevented by Github since it handles out of sync commits thanks to it being an actual git repository. On Github, if a pushed head already exists as a commit it won’t overwrite a newer head, but there are no controls like this in S3. The impact mostly seems to be for new projects.”<p>That is a relatively well known failure pattern with git on s3 and it’s a shame the s3 representatives weren’t aware or there isn’t a knowledge base to tap into. I have found with AWS you have to get beyond the first level of support to have any chance of having a meaningful conversation.<p>When it comes to user data, you have to be extremely careful, so I was surprised on X how quickly they were moving things off of GitHub in a relatively celebratory manner. They had a ton of attention from this incident so I guess they wanted to recover as fast as possible versus being sure they couldn’t lose user data (and to be fair it seems like it was all recoverable). Still should be careful as their service becomes more popular and important.
regardless of scale there will most likely be code paths where performance is not prioritized, and unexpected use-cases can put the stability at risk. In this case that was creating new repos. I believe if we only made a crazy amount of commits there would have been no issue.
To the devs: if I type "<a href="https://lovable.dev/projects/zzz" rel="nofollow">https://lovable.dev/projects/zzz</a>" as the URL, then the browser will start continuously DDOSing the web server with retries. Currently I'm at 2k requests on the page.
Founder here:<p>Not part of the report was that getting intro to people on Github significantly helped accelerate things.<p>Any questions about Lovable infra for those curious generally – happy to answer.<p>(also, we're hiring senior platform engineers in Europe to build the first European AI outlier success story:)