Also:
1) Maximum # of open file descriptors<p>2) Whether your slave DB stopped replicating because of some error.<p>3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings.<p>4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory.<p>5) If you're running out of space in a specific partition you usually store random stuff like /var/log.<p>I've had my ass bitten by all of the above :)
Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report.<p>One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available for applications to use.<p>Another often-overlooked metric that's important for web services in particular is the TCP accept queue depth, per listening port. Once the accept queue is drained, remote clients will get ECONNREFUSED, which is a bad place to be. This value is somewhat difficult to attain, though, because AFAIK Linux doesn't expose it.
Interestingly, an out-of-the-box Munin configuration on Debian contains nearly all of these. I recommend setting up Munin and having a look at what it monitors by default, even if you don't intend to use it as your monitoring solution.
The corollary of this post is "things we've been monitoring and/or alerting on which we shouldn't have been".<p>Starting at a new shop, one of the first things I'll do is:<p>1. Set up a high-level "is the app / service / system responding sanely" check which lets me know, from the top of the stack, whether or not everything else is or isn't functioning properly.<p>2. Go through the various alerting and alarming systems and generally dialing the alerts <i>way</i> back. If it's broken at the top, or if some vital resource is headed to the red, let me know. But if you're going to alert based on a cascade of prior failures (and DoS my phone, email, pager, whatever), then STFU.<p>In Nagios, setting relationships between services and systems, for alerting services, setting thresholds appropriately, etc., is key.<p>For a lot of thresholds you're going to want to find out <i>why</i> they were set to what they were and what historical reason there was for that. It's like the old pot roast recipe where Mom cut off the ends of the roast 'coz that's how Grandma did it. Not realizing it was because Grandma's oven was too small for a full-sized roast....<p>Sadly, that level of technical annotation is often lacking in shops, especially where there's been significant staff turnover through the years.<p>I'm also a fan of some simple system tools such as sysstat which log data that can then be graphed for visualization.
Be sure to monitor your monitoring system as well (preferably from outside your network/datacenters)! If you don't have anything else in place, you can use Pingdom to monitor one website/server for free [0].<p>I was off work for a few months recently (motorcycle wreck) and removed my e-mail accounts from my phone. Now, I have all my alerts go to a specific e-mail address and those are the only mails I receive on my phone. It has really helped me overcome the problem of ignoring messages.<p>[0]: <a href="https://www.pingdom.com/free/" rel="nofollow">https://www.pingdom.com/free/</a>
We monitor outgoing smtp and http connections from anything that requires those services.<p>And the best general advice I have is split your alerts into "stuff that I need to know is broken" and "stuff that just helps me diagnose other problems". You don't want to be disturbing your on-call people for stuff that doesn't directly affect your service (or isn't even something you can fix).
We had a perfect storm of problems only 2 weeks ago.<p>1. A vendor tomcat application had a memory leak, consumed all the RAM on a box, and crashed with an OOM<p>2. The warm standby application was slightly misconfigured, and was unable to take over when the primary app crashed<p>3. Our nagios was configured to email us, but something had gone wrong with ssmtp 2 days prior, and was unable to contact google apps<p>3a. No one was paying any attention to our server metric graphs / We didn't have good enough "pay attention to these specific graphs because they are currently outside the norm"<p>A very embarrassing day for us that one.<p>We're now working on better graphing, and have set up a basic ssmtp check to SMS us if there is an issue. Monitoring is hard.
You're using icanhazip.com in production? I see from a quick Google search that Puppy Linux seems to use it in some scripts, but how reliable is it?
About reboot monitoring, I suggest to use kdump to dump the oops information and save it for later debugging and understanding of the issue. It may even be an uncorrectable memory or pcie error you are seeing and the info is logged in the oops but is hard to figure otherwise. Also, if you consistently hit a single kernel bug you may want to fix it or workaround it.
I have gear in three different facilities and I'm typically visiting any of them unless I'm installing hardware or replacing it. Shortly after starting at $job, I realized there was no monitoring of the RAID arrays in the servers we have. That could have ended badly.