Mistake 3: Putting different people in charge of alert-setting than alert-responding<p>In the best case, it increases the risk of alarm-fatigue. In the pathological cases, it means the technical system becomes a cross-group battleground for shifting blame and workload.
Good, though I find Rob Ewashuk's "My Philosophy on Alerting" essay (Google SRE) better:<p><a href="https://docs.google.com/a/gravitant.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview?sle=true&pli=1#heading=h.fs3knmjt7fjy" rel="nofollow">https://docs.google.com/a/gravitant.com/document/d/199PqyG3U...</a><p>Particularly:<p>Pages [alerts] should be urgent, important, actionable, and real.<p>They should represent either ongoing or imminent problems with your service.<p>Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring.<p>You should almost always be able to classify the problem into one of: availability & basic functionality; latency; correctness (completeness, freshness and durability of data); and feature-specific problems.<p>Symptoms are a better way to capture more problems more comprehensively and robustly with less effort.<p>Include cause-based information in symptom-based pages or on dashboards, but avoid alerting directly on causes.<p>The further up your serving stack you go, the more distinct problems you catch in a single rule. But don't go so far you can't sufficiently distinguish what's going on.<p>If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical.