The following link shows how we do Chaos Engineering in TiDB, an open source distributed database:<p><a href="https://www.pingcap.com/blog/chaos-practice-in-tidb/" rel="nofollow">https://www.pingcap.com/blog/chaos-practice-in-tidb/</a><p>Regarding the Fault Injection tools we are using:<p>- Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers.<p>- SystemTap, a scripting language and tool diagnose of a performance or functional problem.<p>- Fail, gofail for go and fail-rs for Rust<p>- Namazu: a programmable fuzzy scheduler to test a distributed system.<p>We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage