I want to be able to find performance bottlenecks after usage spikes, so I would like a tool which monitors and logs cpu, memory, disk io over time on each of my servers. These are ubuntu servers.<p>If you have any other suggestions for tools I can use to help find bottlenecks, I'd appreciate that too; I'm new to this.
There's a ton of projects, none of which is truly easy to deploy, all of which kinda suck, and all of which store data as RRDs (which is the worst possible solution except for all the others, as the saying goes).<p>This is an area that could use some new blood, but apparently the existing solutions work barely well enough.
You probably know about sar, but a lot of people may not; it can get advanced with all the options, but for simple cpu spot checks:<p><pre><code> sar -u X Y
</code></pre>
prints Y lines of utilization info, one every X seconds.
<a href="http://www.solarisinternals.com/wiki/index.php/Dim_STAT" rel="nofollow">http://www.solarisinternals.com/wiki/index.php/Dim_STAT</a><p><quote>
<i>All STAT data are collected from standard Solaris or Linux programs (vmstat, iostat, etc.) or some special (like psSTAT for users/processes activity) and saved in MySQL database. Collected data are accessed via Web interface and can be presented in several manner (interactive or static graphs, text, HTML tables).</i><p><i>dim_STAT can be used for On-Line monitoring one or several hosts on the same time. As well, data may be easily post loaded from output files of stat commands and analyzed in the same manner. At any time collecting from new stat commands may be added to the tool (via Add-On interface) and enlarge your view on application workload, RDBMS, your personal STAT program, etc.</i>
</quote>
Where I'm working at the moment we use Zabbix to monitor system availability and collect performance and trend data for 184 servers [that generate 8701 monitored items of data].
The Zabbix folk do all their development on Ubuntu and it's their preferred platform; but it works just as well on Redhat, CentOS, etc and has client agents for a wide variety of platforms.
I keep munin, <a href="http://munin.projects.linpro.no/" rel="nofollow">http://munin.projects.linpro.no/</a> installed on all my servers. When things begin to look unwell, it lets you look back to see what resource, temperature, voltage, or activity might have changed.<p>It is alleged to be easy to add your own data points to it as well.
I use zabbix over ~<1000 physical and virtual hosts for historical resource usage and alerting.<p>Cacti for network traffic monitoring (and alerting using thold plugin).<p>Smokeping for network latency and availability monitoring and alerting.