Like many others, this was the wakeup call I needed to download go1.18beta1 and start writing some fuzz tests. I unleashed it upon my JSON log viewer, which has the goal of never suppressing a line even in the face of egregious parse errors. (The last thing you need when software goes wrong is your UI hiding stuff from you, after all.)<p>As expected, within seconds it found that the input `{"": 0}` causes a panic. (This is a weird special case that I handled for other classes of this problem, but not this exact case!) I also got to see what happens when you have more bytes on a line than bufio.MaxScanTokenSize and realized that that is pretty low for an interactive program, especially when all you can do is exit.<p>The ergonomics are excellent. When the fuzzer finds a failing input, it creates a file that contains the test case. You check this in and future invocations of plain "go test ./..." will use this input. "go test -fuzz=FuzzWhatever what/ever" will cause it to search for more failing inputs. Really well-designed.
Fuzzing is awesome. I just discovered an accidental O(2^n) code path in my project with fuzzing and fixed it: <a href="https://github.com/elves/elvish/commit/9cda3f643efafce2df5671bbdd609b11b4b910d5" rel="nofollow">https://github.com/elves/elvish/commit/9cda3f643efafce2df567...</a><p>Edit: shortly after I wrote this comment, fuzzing discovered another pathological input - and that was fixed in <a href="https://github.com/elves/elvish/commit/04173ee8ab3c7fc4a9e793f70a1e3b58b82d3728" rel="nofollow">https://github.com/elves/elvish/commit/04173ee8ab3c7fc4a9e79...</a><p>(In case people are curious, the project is a Unix shell, Elvish: <a href="https://elv.sh" rel="nofollow">https://elv.sh</a>)
Great to see fuzzing becoming more mainstream. Ultimately we have absurd program states, with even a trivial program's state vastly exceeding the number of particles in the universe. We need to start finding order-of-magnitude-better approaches for testing.<p>I almost always write generated tests at this point with unit tests being a fallback for slow code or niche cases. What I <i>dont</i> generally write though is fuzz tests, which would really be a 'next step'. In Rust it's not very hard to do so, but it hasn't quite hit the "trivial" mark yet for me, whereas quickcheck is virtually the same amount of work to use as to not use.<p>Languages like Go adopting and mainstreaming these practices will be a benefit to everyone.<p>I'm curious if there's documentation on:<p>a) The coverage approach taken<p>b) The mutation approach taken<p>Can you configure these? Plugin different fuzzing backends?
Anyone seen good articles on converting go-fuzz tests to native fuzzing? Specifics on the new corpus format and a converter from go-fuzz would be really useful.<p>It’s great to hear that the fuzzer is built on go-fuzz so hopefully the conversion process won’t be too bad: <a href="https://github.com/dvyukov/go-fuzz/issues/329" rel="nofollow">https://github.com/dvyukov/go-fuzz/issues/329</a>
Past related thread:<p><i>Go: Fuzzing Is Beta Ready</i> - <a href="https://news.ycombinator.com/item?id=27391048" rel="nofollow">https://news.ycombinator.com/item?id=27391048</a> - June 2021 (53 comments)
What would you say are the main differences between a fuzzer and QuickCheck? The authors of quickcheck don't call it a fuzzer so I assume there is some difference but both seem to randomize inputs?
I guess I'm echoing others here, but fuzzing is magical. I've given introductions to fuzz-testing to more than a few developers, and found interesting bugs in my own code using them.<p>Until now I've used "go-fuzz", but I'm very much looking forward to having real integrated fuzzing in the standard compiler/toolchain. That has to make things easier to explain and add to more projects.<p>Even with "100% test coverage" finding bugs due to fuzzing shows how hard testing can be.
I will never understand why this has been included in the standard library instead of as a standalone library available for download. Now it's locked to the Go release cycle and have the potential to languish because of backward compatibility concerns.<p>The decision to include it is perplexing when other language ecosystems have chosen to keep this kind of functionality out of the standard lib, e.g. requests in python[1]. To quote Kenneth Reitz: "...the standard library is where a library goes to die."<p>[1] <a href="https://github.com/psf/requests/issues/2424" rel="nofollow">https://github.com/psf/requests/issues/2424</a>