Being the victim of a DDoS attack is never fun but on the other hand it’s a great way to learn something new in the heat of the moment so to speak.
The only way I know of Really stopping a DDoS attack requires access to the BGP implementation used in your AS to at the edge stop the traffic before it reaches the web host in this case via a “blackhole route”.
With that said there are things we can do to mitigate the problem.
The Attack
Earlier today a WordPress site hosted on a CentOS based server running Virtualmin got attacked on the /xmlrpc.php file, the mass query lead to the server spawning hundreds of php-cgi instances resulting in a CPU usage of 100% == The server wasn’t happy. Read more about the xmlrpc.php file and what it’s used for here.
Troubleshooting
First and foremost when faced with a 100% CPU situation is to determine What is hogging all that CPU, with the high CPU usage the Virtualmin management will most likely be very slow, but get it going and access the server via SSH in a separate window.
Run the top command via SSH.
Tasks: 180 total, 2 running, 177 sleeping, 1 stopped, 0 zombie %Cpu(s): 99.0 us, PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19760 mysql 20 0 1091764 28656 0 S 7.3 1.4 0:35.85 mysqld 546 website+ 20 0 321372 29912 19972 S 3.0 1.5 0:00.52 php-cgi .. 550 website+ 20 0 321116 29716 20040 S 2.0 1.5 0:00.42 php-cgi 550 website+ 20 0 321116 29716 20040 S 2.0 1.5 0:00.42 php-cgi 31882 apache 20 0 537480 15540 11992 S 0.3 0.8 0:00.04 httpd
In this example the website+ user is running a bunch of php-cgi instances, causing the CPU usage to go through the roof, the username will hopefully indicate what virtual server is under attack as it’ll match the virtual server admin account.
First priority with this information is to use the Disable feature of Virtualmin to stop virtual server with the result of stopping the flood and in turn lessening the CPU usage allowing all other virtual servers to run as intended again.
Hopefully the CPU usage should with the attacked server disabled subside to normal levels.
Logs
With the newly acquired breathing room it’s time to dig through the logs, Virtualmin provides a GUI for this via Virtualmin > Logs and Reports > Apache Access/Error Log.
1.2.3.4 - - [17/Jul/2017:15:13:16 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 1.2.3.4 - - [17/Jul/2017:15:13:16 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 1.2.3.4 - - [17/Jul/2017:15:13:18 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 1.2.3.4 - - [17/Jul/2017:15:13:18 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 1.2.3.4 - - [17/Jul/2017:15:13:19 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 1.2.3.4 - - [17/Jul/2017:15:13:19 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
Or by SSH,
[admin@server1 ~]# tail -f /var/log/virtualmin/website.com_access_log
In this example we can see that the host 1.2.3.4 is DDoSing the site via xmlrpc.php.
Solution
.htaccess
There are multiple solutions to this problem in particular, if xmlrpc.php is not used at all a quick solution to the problem would be to straight up block access to it via .htaccess as described here, or by using the Disable XML-RPC plugin, YMMW.
# Disable access to xmlrpc.php <Files xmlrpc.php> Order allow,deny Deny from all </Files> # END Disable access to xmlrpc.php
This does require FTP access, can be harmful if not done carefully and is a destructive fix in the sense that it stops functionality, however, if xmlrpc.php is only used by a static set of IP’s it’s a very good solution to explicitly allow those IP’s and block everything else via the .htaccess file, this should help.
iptables
What you could do is close the door on the IP range attacking you via iptables. This however is also a very manual solution and does require administrative assistance, on the positive side it retains functionality for legit users.
As a bonus, blocking the range via iptables is quick- and resource efficient.
sudo iptables -A INPUT -s 1.2.3.0/24 -j DROP
With this example we’ll block Anything coming from 1.2.3.0/24 aimed at Any of our interfaces using Any port, == block the range entirely.
fail2ban
For the meat and potatoes then, fail2ban, a software package that acts as a bouncer for your server.
It operates by scanning log files, for example the _access_log from above, and executes commands based on REGEX matches.
With the help of fail2ban and Virtualmin we’ll set up the service to create a temporary iptables rule in the style of the one above, but only if a specific IP has attempted to load the xmlrpc.php document X times within X minutes (5 times within 10 minutes as defaults).
fail2ban is not loaded with the default installation and has to be manually installed, enabled and configured.
Installation
Firstly the package will have to be installed,
# RPM yum update -y && yum install epel-release -y yum install fail2ban -y # APT apt-get update -y && apt-get upgrade -y apt-get install fail2ban -y
With that done the module should be enable-able with the Webmin GUI.
Webmin > Un-used Modules > Fail2Ban Intrusion Detector, once activated it’ll move from Un-used Modeuls to Networking.
Remember to set the module to start at boot.
Figuring out How fail2ban works will have to be an exercise for another time, you’ll need 3 things to set up the protection;
Log filter: /etc/fail2ban/filter.d/apache-xmlrpc.conf
Match Action: /etc/fail2ban/action.d/apache-xmlrpc.conf
Jail: /etc/fail2ban/jail.conf
Restart the service once configured. service fail2ban-server restart.
Enable the server
Enable the virtual server again via Virtualmin > Disable and Delete > Enable Virtual Server.
If this worked you should see a whole bunch of action in the /var/log/fail2ban.log logfile, provided you are still under attack, as well as a new iptables section.
[admin@server1 ~]# tail -f /var/log/fail2ban.log 2017-07-17 17:15:13,854 fail2ban.actions [22284]: NOTICE [apache-xmlrpc] Ban 1.2.3.16 2017-07-17 17:15:14,077 fail2ban.actions [22284]: NOTICE [apache-xmlrpc] Ban 1.2.3.14 2017-07-17 17:15:14,464 fail2ban.filter [22284]: INFO [apache-xmlrpc] Found 1.2.3.20 2017-07-17 17:15:20,488 fail2ban.filter [22284]: INFO [apache-xmlrpc] Found 1.2.3.21 2017-07-17 17:15:21,314 fail2ban.actions [22284]: NOTICE [apache-xmlrpc] Ban 1.2.3.21 2017-07-17 17:15:22,499 fail2ban.filter [22284]: INFO [apache-xmlrpc] Found 1.2.3.22 2017-07-17 17:15:22,500 fail2ban.filter [22284]: INFO [apache-xmlrpc] Found 1.2.3.20 2017-07-17 17:15:22,536 fail2ban.actions [22284]: NOTICE [apache-xmlrpc] Ban 1.2.3.22 2017-07-17 17:15:28,523 fail2ban.filter [22284]: INFO [apache-xmlrpc] Found 1.2.3.20 2017-07-17 17:15:28,770 fail2ban.actions [22284]: NOTICE [apache-xmlrpc] Ban 1.2.3.20
[admin@server1 ~]# iptables -L INPUT -v -n Chain INPUT (policy ACCEPT 162 packets, 31256 bytes) pkts bytes target prot opt in out source destination 54786 5970K f2b-apache-xmlrpc-http tcp -- * * 0.0.0.0/0 0.0.0.0/0
Fail2Ban is now doing what it’s designed to do, blocking any attack attempts while leaving legit traffic untouched, are there nobs to tweak, optimizations to be done, configuration to be had? Read more here.
Not only does this protect the affected site, but it does protect every virtual server running on the host.
With the same kind of thought process it’s possible to protect the /wp-login.php page aswell, there are some hints available at the GitHub.