This article was absolutely painful to read. Let me try a different ending:<p>> Conclusion:<p>1. Ensure the files that php is running are not writable by the same process. Different app -> different user.<p>2. Unless you're planning to send emails from the server, firewall output on those ports. If you do plan to, firewall everything apart from that server (you can setup alerts when DNS changes, it's not going to happen often for email hosts)<p>3. Disable most unnecessary functions and modules. Anything touching eval, str_rot13, exec, and many others should be killed right away.<p>4. Enable basedir.<p>5. If you can't handle regular system updates, don't run your own server. If you can't handle wordpress updates, host it with someone else.<p>6. Wherever you host, make sure your app can be easily redeployed. You can't rely on dates to see when anything changed.<p>7. If you can't handle system upgrades, don't think that docker or any software is going to solve any of your problems.
In short: an old server running un-updated software was broken into not once but at least four times since the start of the year.<p>The post describes some steps the author took to investigate and block the attack. It's an entertaining reading, but I'd strongly advise against trying this at home: it is not worth the risk. Reinstalling the system from scratch instead would have been much more prudent.
Fun read. If you ever get hacked, I recommend just destroying the server and starting from scratch and a clean backup.<p>If you try to find all backdoors and left over rootkits, you will end up forgetting one and being re-compromised.
One may find the following resources useful for measures on how to secure one's infrastructure:<p>- Security Guide: How to Protect Your Infrastructure Against the Basic Attacker [0]<p>- 7 Security Measures to Protect Your Servers [1]<p>[0] <a href="http://blog.mailgun.com/security-guide-basic-infrastructure-security/" rel="nofollow">http://blog.mailgun.com/security-guide-basic-infrastructure-...</a><p>[1] <a href="https://www.digitalocean.com/community/tutorials/7-security-measures-to-protect-your-servers" rel="nofollow">https://www.digitalocean.com/community/tutorials/7-security-...</a>
Question about this:<p>It's not possible for me to track all the 0 days for every piece of software and library that my servers run. One of my long running servers could have been backdoor-ed by a 0 day 6 months ago and i probably wouldn't know it. The servers are kept updated but 0 days don't care about that by definition.<p>What's the best practice here? Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them?
You should consider running this malware detection script[0]. That script is designed to catch malware php scripts. I've used it before with lots of success. You will also want to check the access logs for all php scripts hit in the last few months. Your hacker's spam script is likely being activated/ran by a GET or a POST to that script. That's a pretty cheap way to screen for other compromised files.<p>[0] - <a href="https://www.rfxn.com/projects/linux-malware-detect/" rel="nofollow">https://www.rfxn.com/projects/linux-malware-detect/</a>
I did get one of these OVH spam warning emails too once. It was for a failover mail server. It turned out that the software I was using (smartermail) wasn't using TLS by default so the link between the failover and the primary server was unencrypted. Spammers often target failover over primary servers. And OVH was interpreting the failover passing on the spam back to the primary server as me sending spam.
Don't delete the outbound mail queue as they have a header in them which tells you which php script sent them.<p>Also in the httpd logs look for POST 200's to scripts that shouldn't be getting them, and look for long query strings esp with base64 in them. Check those scripts and also scripts messed with by the same user-agent and ip's.<p>Never just delete anything, always check the contents and write a script that can search and nuke it en masse. e.g. Find | xargs | sed etc.<p>When I had issues earlier this year I had postfix outbound throttled right down, and a script to shut it down completely if the queue got big. So if I got the monitoring email about smtp going down I'd know I'd missed something ;)<p>Ps, wp-cli, you need it in your life :)
Thanks for sharing your intrusion story. Although it is always painful for the person to experience, it gives good insights on what happens in the wild. As the author of security tools rkhunter and Lynis, it is still a sign that basic (and advanced) system hardening remains needed for a long time.<p>The other comments show that there are so many things you could do. But some are more relevant than others. That is something we want to test for in Lynis with upcoming updates. One focus area will be detecting Drupal/Joomla/WordPress installations. If anyone wants to help out the open source project and make the web a safer place, this is your chance to help: <a href="https://github.com/CISOfy/lynis" rel="nofollow">https://github.com/CISOfy/lynis</a><p>Thanks again for sharing!
I wish it was easier to convert "standard" dynamic web pages into static HTML easier. There's one or two Wordpress sites I'd really like to get rid of updating over and over again (which is still less work than actually reading mostly meaningless Changelogs and making a decision if there are security issues that actually apply to me). Probably, my solution to this problem will be "wget -r", plus a bit of sanitizing.
Does }__ appear in your logs? All versions of all branches of Joomla prior to I think 3.4.6 had a problem with serialization that allowed arbitrary PHP execution.
Here are my advice:<p>1. once compromized, start with a fresh copy. Use ansible to define automatic installation scripts so that the server can be fully reinstalled in a "snap". Use backups of the user data to restore.<p>2. Automate upgrades. It's a piece of cake with ansible.<p>3. Stay away from php. There are too many potential loopholes. I now start using go to implement my web servers. Especially if the web site is static. Check Hugo or tools like that. I implement dynamic servers with go and Iris that uses fasthttp.