There I was, minding my own business… Coding… drinking coffee… and you know… working 🙂
Then some hackers came along and ruined all the fun. Digitalocean were fast to take action and closed all the eth-interfaces on the server and contacted me.
Fortunately it’s a redundant setup, so closing down one server didn’t cause any issues for our customers.
It did however, leave me with a great deal of work in order to fix the server again.
I never reuse a server that has been hacked. “Delete it, create a new, and rethink the solution to make it better”.
The server only exposes ssh and a webserver to the world, so one of those would have had to be the target.
From what I could see in various logs, they (the hackers) simply tried a million logins to get into the server, using ssh.
So if brute forcing their way in, is the only way they can think of, we might as well ban their IPs.
I found a nice little opensource project called fail2ban which installs nicely on all major linux distros, using the packagemanagers. (which means no compiling and dependency management and manually updating the script for me!)
It simply parses the /var/log/auth.log file (on ubuntu/debian, and /var/log/secure.log on other distros) and sees if a certain IP has been trying too many times to signin without success. (well, this is the default SSH behaviour at least).
However, on ubuntu I found a problem with the auth.log file, since is grouped redundant messages.
So if fail2ban tests for multiple matching lines, and ubuntu groups similar lines, then it wouldn’t work… and it didn’t.
It was an easy fix though.
Simply edit /etc/rsyslog.conf and find “$RepeatedMsgReduction on”. Set this to “off”, save the file, restart the service (sudo service rsyslog restart) and you are good to go.
fail2ban will now start to parse the log file (/var/log/auth.log) and little by little the attackers are banned in the firewall.
Simple and efficient.
The default settings is to ban the attackers for 10mins after 3-6 tries.
Now you might ask: “why not just ban them forever?”… well, because I sometimes type passwords wrong 🙂
I use randomly generated passwords and I could easily hit 3 tries and get banned. So banning IPs for 10mins seems like a fair solution 🙂
And remember kids, always update the software on your servers at least once a week 🙂
A lot of critical errors are fixed all the time, and if you install your apps using the packagemanager, it doesn’t take more than a few mins per server.