Hi HN, we are Spriha and Ankit building Flakybot is a tool to automatically identify and suppress test-flakiness so that developers are better able to trust their test results.<p>Most CI systems leave it up to teams to manually identify and debug test flakiness. Since most CI systems today don’t handle test reruns, teams just end up with manually rerunning tests that are flaky. Ultimately, tribal knowledge gets built over time where certain tests are known to be flaky, but the flakiness isn’t specifically addressed. Our solution, Flakybot, removes one of the hardest parts of the problem: identifying flaky tests in the first place.<p>We ingest test artifacts from CI systems, and note when builds are healthy (so that we can mark them as “known-good builds” to use while testing for flakiness). This helps automatically identify flakiness, and proactively offer mitigation strategies, both in the short term and long term. You can read more about this here:
<a href="https://ritzy-angelfish-3de.notion.site/FlakyBot-How-it-works-5ab7e20ca01e4efc93732897501e52b4" rel="nofollow">https://ritzy-angelfish-3de.notion.site/FlakyBot-How-it-work...</a><p>We’re in the early stages of development and are opening up Flakybot for private beta to companies that have serious test-flakiness issues. The CI systems we currently support are Jenkins, CircleCI and BuildKite, but if your team uses a different CI and has very serious test-flakiness problems, sign up anyway and we’ll reach out. During the private beta, we’ll work closely with our users to ensure their test flakiness issues are resolved before we open it up more broadly.
How about you fix the flaky tests? Am I insane for thinking that? The whole concept of "just reboot it" or "re run it again" and "fixing" the problem is at least one reason the modern world sits on a mountain of complete garbage software.
What I want is a tool to make flaky tests fail reliably.<p>They won't be fixed until they start actually preventing commits. If somebody deletes a test, that is on that person. I don't want a tool automatically suppressing testing.
Congrats on getting Flakybot into private beta!<p>For people that prefer to minimize the number of moving parts, CircleCI now also has built-in Flaky test detection:<p><a href="https://circleci.com/blog/introducing-test-insights-with-flaky-test-detection/" rel="nofollow">https://circleci.com/blog/introducing-test-insights-with-fla...</a><p>Of course this is coupled to our CircleCI platform, so if you want to stay agnostic for sure check out Flakybot :)<p>NB I work for CircleCI
Tests can fail for a multitude of "flakey" reasons, that aren't due to 'source code' changes.<p>- Network connections failed<p>- Run out of memory<p>- Duration timeouts<p>- Setting up the test infrastructure failed<p>All these can contribute to flakiness, and could be detected and reported by a bot like this. Making this a useful idea!<p><i>"Your test failed, but it was consider a flakey fail [Network Flakiness]" - FlakyBot</i>
We've been relying on manual testing so far. We're just starting to think about unit tests and integration tests. We don't know where to start. Would be cool if you could provide guidance on setting up good testing practices in the first place so that we avoid flaky tests all together.
Nice. I’m for us building a rerun bot and I’m trying to advocate for it at work but not everyone agrees. Nice to pull this out as a service. Well done.