Disclosure: I work on Google Cloud.<p>I'm super pleased to see this! Abhishek and the cluterfuzz team were one of our initial customers for Preemptible VMs, still are, and make for a great example. Congrats to the team!
I don't want to hijack the thread subject but here are my thoughts on the usefulness of fuzzing of safe languages.<p>Even in the absence of memory corruption bugs there is a subclass of bugs that can emerge in any general-purpose language, like slowness/hangs, assert failures, panics and excessive resource consumption.<p>Barring those, you can detect invariant violations, (de)serialization inconsistencies (eg. deserialize(serialize(input)) != input, eg. see [1]), different behavior across multiple libraries whose semantics must be identical (crypto currency implementations are notable in this regard as deviation from the spec or canonical implementation in the execution of scripts or smart contracts can lead to chain splits).<p>With some effort you can do differential 64 bit/32 bit fuzzing on the same machine, and I've found interesting discrepancies between the interpretation of numeric values in JSON parsers, which makes sense if you think about it (size_t and float have a different size on each architecture, causing the 32 bit parser to truncate values). This might be applicable to every language that does not guarantee type sizes across architectures like Go (not sure?), but I haven't tested that yet.<p>You can detect path escape/traversal (which is entirely language-agnostic but potentially severe) by asserting that any absolute path that is ever accessed within an app has a legal path, or by fuzzing a path sanitizer specifically.<p>And so on.<p>Code coverage is the primary metric used in fuzzing, but other metrics can be useful as well. I've experimented extensively with metrics such as allocation, code intensity (number of basic blocks executed) (which helped me prove that V8's WASM JIT compiler can be subjected to inputs of average size that take >20 seconds to compile), and stack depth, see also [2].<p>Any quantifier can be used as a fuzzing metric, for example the largest difference between two variables in your program.<p>Let's say you have a decompression algorithm that takes C as an input and outputs D. Calculate R = len(D) / len(C), so that R is the ratio between compressed input and decompressed output. Use R as a fuzzing metric and the fuzzer will tend to generate inputs that have a high compressed/decompressed size ratio, possibly leading to the discovery of decompression bombs [3].<p>Wrt. this, libFuzzer now also natively supports custom counters I believe [4].<p>Based on Rody Kersten's work I implemented libFuzzer-based fuzzing of Java applications supporting code coverage, intensity and allocation metrics [5], and it should not be difficult to plug this into ClusterFuzz/oss-fuzz.<p>Feel free to get in touch if you have any questions or need help.<p>[1] <a href="https://github.com/nlohmann/json/blob/develop/test/src/fuzzer-parse_json.cpp" rel="nofollow">https://github.com/nlohmann/json/blob/develop/test/src/fuzze...</a><p>[2] <a href="https://github.com/guidovranken/libfuzzer-gv" rel="nofollow">https://github.com/guidovranken/libfuzzer-gv</a><p>[3] <a href="https://en.wikipedia.org/wiki/Zip_bomb" rel="nofollow">https://en.wikipedia.org/wiki/Zip_bomb</a><p>[4] <a href="https://llvm.org/docs/doxygen/FuzzerExtraCounters_8cpp_source.html" rel="nofollow">https://llvm.org/docs/doxygen/FuzzerExtraCounters_8cpp_sourc...</a><p>[5] <a href="https://github.com/guidovranken/libfuzzer-java" rel="nofollow">https://github.com/guidovranken/libfuzzer-java</a>
Thank you for open sourcing this. For those interested in trying multiple cluster-based fuzzing solutions, I'd also like to point at yahoo/yfuzz[1], which is k8s-backed.<p>[1] <a href="https://github.com/yahoo/yfuzz" rel="nofollow">https://github.com/yahoo/yfuzz</a>
For those interested in the repo: <a href="https://github.com/google/clusterfuzz" rel="nofollow">https://github.com/google/clusterfuzz</a>
Is there a fuzzing tool oriented towards web applications? Something that could generate loads of Selenium cases automatically and verify whether the application crashes, logs an exception or continues to work smoothly??
Makes you think about choosing to write software in C / C++ / other non-memory-safe languages when you need 25000 cores churning away to ensure you don’t make mistakes that could cause serious security issues.<p>It makes me wonder why Google wouldn’t put their efforts into using Rust, for example.<p>Of course, server power is cheap, but not for our planet.