We created the first open source implementation of Meta's TestGen–LLM

How do people feel about LLM generated tests?<p>I tried creating some on a personal project just using ChatGPT and it saved me a lot of toil on tests I probably wouldn’t have written. I did find I had low trust in refactoring my code, but higher than if I’d had no tests.<p>It seemed like a net positive for low risk cases.

Per the cited real world figures, that&#x27;s about 1 in 40 tests that pass human review, or a success rate of about 2.5%.<p>It&#x27;s hard to see value in spending resources this way right now - most notably, engineer time to review the generated tests. Improve the hit rate by an order of magnitude, and I suspect I&#x27;d feel differently.

Tried this out on a Ruby codebase and it generated Python tests: <a href="https:&#x2F;&#x2F;github.com&#x2F;Codium-ai&#x2F;cover-agent&#x2F;issues&#x2F;17">https:&#x2F;&#x2F;github.com&#x2F;Codium-ai&#x2F;cover-agent&#x2F;issues&#x2F;17</a>. Is there any data available on whether this actually works?

Why does this webpage have auto-playing audio?

The audio track on load that has no obvious way to stop playing prevents me from reading this content. Please don&#x27;t do that.

Using ChatGPT to generate unit tests works great almost out of the box, but I guess this system solves the remaining 5% to make it fully automated end-to-end. I believe this will work and help us write better software, given that I have experienced numerous cases where the generated tests (even with inferior models) catch no-so-obvious bugs.

Seems decent enough for boilerplate. But if my code is incorrect, won’t an LLM generated a test for incorrect code?

Interesting idea. I generally don’t run tests at all (hobbyist) so even mediocre llm tests may actually be a win

Don&#x27;t see any actual output measurement in the conclusion — it seems like the effort may not have really borne fruit.

To the OP:<p>Is your name a reference to Gronky Scripples? <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=4KG3v365mq4" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=4KG3v365mq4</a>

Love that you took something that meta wrote about but didn&#x27;t actually release and then... did it for them haha :)

I get redirected to an oops 404 page when I try to create an account using Github.

Any chance of supporting integrations with AWS, Azure, GCP APIs?