instagram arrow-down
Kalle Lilja

Archives

Protect your WordPress site from xmlrpc.php attacks

Running on Virtualmin using fail2ban

Being the victim of a DDoS attack is never fun but on the other hand it’s a great way to learn something new in the heat of the moment so to speak.
The only way I know of Really stopping a DDoS attack requires access to the BGP implementation used in your AS to at the edge stop the traffic before it reaches the web host in this case via a “blackhole route”.
With that said there are things we can do to mitigate the problem.

The Attack

Earlier today a WordPress site hosted on a CentOS based server running Virtualmin got attacked on the /xmlrpc.php file, the mass query lead to the server spawning hundreds of php-cgi instances resulting in a CPU usage of 100% == The server wasn’t happy. Read more about the xmlrpc.php file and what it’s used for here.

Troubleshooting

First and foremost when faced with a 100% CPU situation is to determine What is hogging all that CPU, with the high CPU usage the Virtualmin management will most likely be very slow, but get it going and access the server via SSH in a separate window.
Run the top command via SSH.

Tasks: 180 total,   2 running, 177 sleeping,   1 stopped,   0 zombie
%Cpu(s):  99.0 us,
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
19760 mysql     20   0 1091764  28656      0 S  7.3  1.4   0:35.85 mysqld
  546 website+  20   0  321372  29912  19972 S  3.0  1.5   0:00.52 php-cgi
  ..
  550 website+  20   0  321116  29716  20040 S  2.0  1.5   0:00.42 php-cgi
  550 website+  20   0  321116  29716  20040 S  2.0  1.5   0:00.42 php-cgi
31882 apache    20   0  537480  15540  11992 S  0.3  0.8   0:00.04 httpd

In this example the website+ user is running a bunch of php-cgi instances, causing the CPU usage to go through the roof, the username will hopefully indicate what virtual server is under attack as it’ll match the virtual server admin account.
First priority with this information is to use the Disable feature of Virtualmin to stop virtual server with the result of stopping the flood and in turn lessening the CPU usage allowing all other virtual servers to run as intended again.
Virtualmin Disable Virtual Server
Hopefully the CPU usage should with the attacked server disabled subside to normal levels.

Logs

With the newly acquired breathing room it’s time to dig through the logs, Virtualmin provides a GUI for this via Virtualmin > Logs and Reports > Apache Access/Error Log.

1.2.3.4 - - [17/Jul/2017:15:13:16 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
1.2.3.4 - - [17/Jul/2017:15:13:16 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
1.2.3.4 - - [17/Jul/2017:15:13:18 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
1.2.3.4 - - [17/Jul/2017:15:13:18 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
1.2.3.4 - - [17/Jul/2017:15:13:19 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
1.2.3.4 - - [17/Jul/2017:15:13:19 +0200] "POST /xmlrpc.php HTTP/1.0" 403 212 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"

Or by SSH,

[admin@server1 ~]# tail -f /var/log/virtualmin/website.com_access_log

In this example we can see that the host 1.2.3.4 is DDoSing the site via xmlrpc.php.

Solution

.htaccess

There are multiple solutions to this problem in particular, if xmlrpc.php is not used at all a quick solution to the problem would be to straight up block access to it via .htaccess as described here, or by using the Disable XML-RPC plugin, YMMW.

# Disable access to xmlrpc.php
<Files xmlrpc.php>
Order allow,deny
Deny from all
</Files>
# END Disable access to xmlrpc.php

This does require FTP access, can be harmful if not done carefully and is a destructive fix in the sense that it stops functionality, however, if xmlrpc.php is only used by a static set of IP’s it’s a very good solution to explicitly allow those IP’s and block everything else via the .htaccess file, this should help.

iptables

What you could do is close the door on the IP range attacking you via iptables. This however is also a very manual solution and does require administrative assistance, on the positive side it retains functionality for legit users.
As a bonus, blocking the range via iptables is quick- and resource efficient.

sudo iptables -A INPUT -s 1.2.3.0/24 -j DROP

With this example we’ll block Anything coming from 1.2.3.0/24 aimed at Any of our interfaces using Any port, == block the range entirely.

fail2ban

For the meat and potatoes then, fail2ban, a software package that acts as a bouncer for your server.
It operates by scanning log files, for example the _access_log from above, and executes commands based on REGEX matches.
With the help of fail2ban and Virtualmin we’ll set up the service to create a temporary iptables rule in the style of the one above, but only if a specific IP has attempted to load the xmlrpc.php document X times within X minutes (5 times within 10 minutes as defaults).

fail2ban is not loaded with the default installation and has to be manually installed, enabled and configured.

Installation

Firstly the package will have to be installed,

# RPM
yum update -y && yum install epel-release -y
yum install fail2ban -y
# APT
apt-get update -y && apt-get upgrade -y
apt-get install fail2ban -y

With that done the module should be enable-able with the Webmin GUI.
Webmin > Un-used Modules > Fail2Ban Intrusion Detector, once activated it’ll move from Un-used Modeuls to Networking.
Remember to set the module to start at boot.
fail2ban virtualmin menu webminstart at boot
Figuring out How fail2ban works will have to be an exercise for another time, you’ll need 3 things to set up the protection;
Log filter: /etc/fail2ban/filter.d/apache-xmlrpc.conf
Match Action: /etc/fail2ban/action.d/apache-xmlrpc.conf
Jail: /etc/fail2ban/jail.conf
Restart the service once configured. service fail2ban-server restart.
fail2ban filter apache-xmlrpcfail2ban match apache-xmlrpc

Enable the server

Enable the virtual server again via Virtualmin > Disable and Delete > Enable Virtual Server.
If this worked you should see a whole bunch of action in the /var/log/fail2ban.log logfile, provided you are still under attack, as well as a new iptables section.

[admin@server1 ~]# tail -f /var/log/fail2ban.log
2017-07-17 17:15:13,854 fail2ban.actions        [22284]: NOTICE  [apache-xmlrpc] Ban 1.2.3.16
2017-07-17 17:15:14,077 fail2ban.actions        [22284]: NOTICE  [apache-xmlrpc] Ban 1.2.3.14
2017-07-17 17:15:14,464 fail2ban.filter         [22284]: INFO    [apache-xmlrpc] Found 1.2.3.20
2017-07-17 17:15:20,488 fail2ban.filter         [22284]: INFO    [apache-xmlrpc] Found 1.2.3.21
2017-07-17 17:15:21,314 fail2ban.actions        [22284]: NOTICE  [apache-xmlrpc] Ban 1.2.3.21
2017-07-17 17:15:22,499 fail2ban.filter         [22284]: INFO    [apache-xmlrpc] Found 1.2.3.22
2017-07-17 17:15:22,500 fail2ban.filter         [22284]: INFO    [apache-xmlrpc] Found 1.2.3.20
2017-07-17 17:15:22,536 fail2ban.actions        [22284]: NOTICE  [apache-xmlrpc] Ban 1.2.3.22
2017-07-17 17:15:28,523 fail2ban.filter         [22284]: INFO    [apache-xmlrpc] Found 1.2.3.20
2017-07-17 17:15:28,770 fail2ban.actions        [22284]: NOTICE  [apache-xmlrpc] Ban 1.2.3.20
[admin@server1 ~]# iptables -L INPUT -v -n
Chain INPUT (policy ACCEPT 162 packets, 31256 bytes)
 pkts bytes target     prot opt in     out     source               destination
54786 5970K f2b-apache-xmlrpc-http  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0

Fail2Ban is now doing what it’s designed to do, blocking any attack attempts while leaving legit traffic untouched, are there nobs to tweak, optimizations to be done, configuration to be had? Read more here.
Not only does this protect the affected site, but it does protect every virtual server running on the host.

With the same kind of thought process it’s possible to protect the /wp-login.php page aswell, there are some hints available at the GitHub.