I have been upping all my security lately at the expense of downtime. The first signs I knew that something was wrong is when my logs were logging to /dev/null, and I had unknown ip routes. There are many tools out there for prevention and analysis (clamscan, nmap, wireshark), but I'd really love to know some methods to KNOW when something IS wrong.
Some tools of note here: ossec, alienvault ossiem, selinux, pf, iptables etc. what you need is a hids, system accounting, and hardened os (proper acls, selinux, upto date binaries, firewalls etc).<p>Log correlation helps too. Oh and chuck in notifications for all the above (e-mail, pager etc.) and I think you should be set for future. But please remember, these are not silver bullet.