Did it have near 100% coverage or did it make tradeoffs on what to test?<p>Did it include UI tests which are notoriously difficult, and if so how did it handle issues like timeouts and the async nature of UI?<p>Did it have rigid separation of concepts between unit vs integration tests etc, or more fluid?<p>Could you refactor internal code without changing tests — the holy grail.
The test suite was 90% "end to end" unit tests - no real infrastructure was used it was all faked. Only interactions with the outside world (web client, LLM, database) were tested and all interactions were faked.<p>(This is not feasible on every project but it was on this one, database interactions were simple)<p>There were a small number (~5%) of slow tests that used a real LLM, database, infrastructure, etc. and a small number of very low level unit tests (~5%) surrounding only complex stateless functions with simple interfaces.<p>Refactoring could be done trivially without changing any test code 98% of the time.<p>Additionally, the (YAML) tests could rewrite their expected responses based upon the actual outcome - e.g. when you added a new property to a rest api response you just reran the test in update mode and eyeballed the test.<p>There was also a template used to generate how-to markdown docs from the YAML.<p>Test coverage was probably 100% but I never measured it. All new features being written with TDD/documentation driven development probably guaranteed it.
I've found the Metabase test suite[0] to be very good considering it's real-world software written by a for-profit company. Coverage is good, the correct tests usually break when doing a refactor (stuff like "Oh, I thought this change was harmless but actually it breaks the permissions model"), etcetera. But the most important thing is that there's a strong team culture of a) demanding good tests on each PR b) hunting down flaky tests and other sources of friction.<p>Another neat thing was that there used to be a full-time SDET who spent a lot of time writing Cypress reproductions for known bugs. When you picked up the bug, you could un-skip the test that was right there waiting for you.<p>All that said, of course it's far from perfect!<p>0: <a href="https://github.com/metabase/metabase/">https://github.com/metabase/metabase/</a> Backend unit tests are in test/, Frontend unit tests are in frontend/test, end-to-end tests (Cypress) are in e2e.