Throwing cores at the problem with `pytest-xdist` is typically the lowest hanging fruit, but you still hit all the paper cuts the authors mention -- collection, DB fixtures, import time, etc.<p>And, further optimization is <i>really</i> hard when the CI plumbing starts to dominate. For example, the last Warehouse `test` job I checked has 43s of Github Actions overhead for 51s of pytest execution time (half the test action time and approaching 100% overhead).<p>Disclosure: Have been tinkering on a side project trying to provide 90% of these pytest optimizations automatically, but also get "time-to-first-test-failure" down to ~10 seconds (via warm runners, container snapshotting, etc.). Email in profile if anyone would like to swap notes.
I don't understand why pytest's collection is so slow.<p>On our test suite (big django app) it takes about 15s to collect tests. So much that we added a util using ripgrep to find the file and pass it as an argument to pytest when using `pytest -k <testname>`.
One thing not mentioned here is putting your test database on a RAM disk, aka tmpfs. This significantly speeds up all DB-related tests that use transactions, fixture loading, and migrations.<p>In most distros, /tmp is mounted as tmpfs, but YMMV.
I generally try to avoid mocking completely. However, speeding up tests is an appropriate use. If someone changes the implementation the mock usually simply doesn't apply and the test still works as intended.<p>For example, a great speed optimization in our tests recently was to mock time.sleep.<p>Why do we have so many sleeps? This is testing a test framework for embedded devices where there is plenty of fiddling-then-wait-for-the-hardware.<p>I also mocked some file system accesses. Unit testing is about our application logic and not about Linux kernel behavior anyways.
Pretty good article, it's really a challenge to properly isolate DB operations during testing so having a difference instance per worker is nice. I remember trying to use different schemas (not instances) but I had a hard time to isolate roles as well.
Is Trail of Bits transitioning out of "crypto"?<p>Imho, they are one of the best auditors out there for smart contracts. Wouldn't be surprising to see some of these talented teams find bigger markets.
I get that pytest has features that unittest does not, but how is scanning for test files in a directory considered appropriate for what is called a high security application in the article?<p>For high security applications the test suite should be boring and straightforward. pytest is full of magic, which makes it so slow.<p>Python in general has become so complex, informally specified and bug ridden that it only survives because of AI while silencing critics in their bubble.<p>The complexity includes PSF development processes, which lead to:<p><a href="https://www.schneier.com/blog/archives/2024/08/leaked-github-python-token.html" rel="nofollow">https://www.schneier.com/blog/archives/2024/08/leaked-github...</a>