{"id":1526,"date":"2015-04-23T11:32:04","date_gmt":"2015-04-23T16:32:04","guid":{"rendered":"http:\/\/swildow.darktech.org\/wp\/?p=1526"},"modified":"2015-04-23T11:32:04","modified_gmt":"2015-04-23T16:32:04","slug":"fix-whmcpanel-cphulk-brute-force-protection-lock-out-via-ssh","status":"publish","type":"post","link":"http:\/\/www.wildow.com\/blog\/?p=1526","title":{"rendered":"Fix WHM\/cPanel cPHulk Brute Force Protection Lock Out Via SSH"},"content":{"rendered":"
\n
\n

Fix WHM\/cPanel cPHulk Brute Force Protection Lock Out Via SSH<\/a><\/h1>\n<\/div>\n<\/div>\n
\n

One of my company\u2019s servers is hosted with\u00a0<\/span>Liquid Web<\/a>. Yesterday one of my co-workers tries to log into\u00a0<\/span>WHM<\/a>\u00a0<\/span>and sees the following message:<\/p>\n

Brute Force Protection<\/strong><\/p>\n

This account is currently locked out because a brute force attempt was detected. Please wait 10 minutes and try again. Attempting to login again will only increase this delay. If you frequently experience this problem, we recommend having your username changed to something less generic.<\/p><\/blockquote>\n

He tries to log in a bit later and receives the message again. He contacts Liquid Web\u2019s Heroic Support, and the support person \u201chelpfully\u201d recommend a server reboot to fix the problem. My co-worker asked for another solution, and support said that it was the only way.<\/p>\n

My co-worker tells me about the situation, and I tell him that the support guy is an idiot. It takes all of about ten seconds to quickly find a solution to the problem on google. I then tell my co-worker that I\u2019ll fix the problem, so he \u201cthanks\u201d the support guy and closes the chat.<\/p>\n

I logged into the box via SSH and had the problem fixed in a couple of minutes.<\/p>\n

Information Gathering<\/h3>\n

Since access to the box wasn\u2019t urgent, I did what I recommend that everyone does in situations like this: gather information first. I wanted to know what caused the lock out first. In other words, who or what was brute forcing the box and caused this issue.<\/p>\n

It was quite possible that my box was still under attack. If I simply turned off brute force protection to bypass the block, I could have opened up my box to being compromised.<\/p>\n

cPHulk stores all of its information in a database called\u00a0<\/span>cphulkd<\/code>. There are two tables of interest:\u00a0<\/span>logins<\/code>\u00a0<\/span>and\u00a0<\/span>brutes<\/code>. The logins table stores login authentication failures. The brutes table stores excessive authentication failures indicative of a brute force attack.<\/p>\n

Here is what I saw on the server:<\/p>\n

[chris@office<\/span> ~<\/span>]$<\/span> ssh server<\/span>\r\nLast login: Wed Oct 14 11:02:14 2009 from host\r\n[user@server<\/span> ~<\/span>]$<\/span> mysql -u user -p<\/span>\r\nEnter password:\r\nWelcome to the MySQL monitor.  Commands end with ; or \\g.\r\nYour MySQL connection id is id\r\nServer version: version\r\n\r\nType 'help;' or '\\h' for help. Type '\\c' to clear the current input\r\nstatement.\r\n\r\nmysql><\/span> connect cphulkd<\/span>\r\nReading table information for completion of table and column names\r\nYou can turn off this feature to get a quicker startup with -A\r\n\r\nConnection id:    id\r\nCurrent database: cphulkd\r\n\r\nmysql><\/span> select IP, BRUTETIME from brutes order by BRUTETIME;<\/span>\r\nEmpty set (0.00 sec)\r\n\r\nmysql><\/span> select IP, LOGINTIME FROM logins order by LOGINTIME;<\/span>\r\n+---------------------------------+---------------------+\r\n| IP                              | LOGINTIME           |\r\n+---------------------------------+---------------------+\r\n| 220.199.6.48                    | 2009-10-14 11:23:10 |\r\n| 220.199.6.48                    | 2009-10-14 11:23:10 |\r\n| 220.199.6.48                    | 2009-10-14 11:23:10 |\r\n| 118.212.186.59                  | 2009-10-14 11:23:40 |\r\n| 118.212.186.59                  | 2009-10-14 11:23:40 |\r\n| 118.212.186.59                  | 2009-10-14 11:23:40 |\r\n| djdeatheater.liquidweb.com      | 2009-10-14 11:24:03 |\r\n| 221.7.58.37                     | 2009-10-14 11:24:07 |\r\n| 221.7.58.37                     | 2009-10-14 11:24:07 |\r\n| 221.7.58.37                     | 2009-10-14 11:24:07 |\r\n| djdeatheater.liquidweb.com      | 2009-10-14 11:24:09 |\r\n| djdeatheater.liquidweb.com      | 2009-10-14 11:24:15 |\r\n| mail.ingener.com                | 2009-10-14 11:24:53 |\r\n| mail.ingener.com                | 2009-10-14 11:24:57 |\r\n| 123.147.144.45                  | 2009-10-14 11:25:16 |\r\n| 123.147.144.45                  | 2009-10-14 11:25:16 |\r\n| 123.147.144.45                  | 2009-10-14 11:25:16 |\r\n| 119.62.128.42                   | 2009-10-14 11:25:41 |\r\n| 119.62.128.42                   | 2009-10-14 11:25:41 |\r\n| 119.62.128.42                   | 2009-10-14 11:25:41 |\r\n| pomme.sai.msu.ru                | 2009-10-14 11:26:13 |\r\n| pomme.sai.msu.ru                | 2009-10-14 11:26:13 |\r\n| pomme.sai.msu.ru                | 2009-10-14 11:26:13 |\r\n| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |\r\n| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |\r\n| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |\r\n| 114.143.242.51                  | 2009-10-14 11:27:23 |\r\n| 114.143.242.51                  | 2009-10-14 11:27:23 |\r\n| 114.143.242.51                  | 2009-10-14 11:27:23 |\r\n| 222.179.116.53                  | 2009-10-14 11:27:47 |\r\n| 222.179.116.53                  | 2009-10-14 11:27:47 |\r\n| 222.179.116.53                  | 2009-10-14 11:27:47 |\r\n+---------------------------------+---------------------+\r\n32 rows in set (0.00 sec)<\/pre>\n
As you can see, a distributed brute force login attempt was launched starting at 11:23am. Fortunately, cPHulk quickly recognized the attack and had completely shut down login access for the user account.<\/p>\n
Thus, when we tried to log in, it of course didn\u2019t work.<\/p>\n
Regaining Access<\/h3>\n
Since I now knew the problem, and now felt good about how cPHulk protected our box, I decided to open up access again so we could go about our work. There are two different ways that I could have done this.<\/p>\n
The first way would be to disable cPHulk to regain access, log into WHM, clear out the the block by using the \u201cFlush DB\u201d option in the cPHulk settings page, and then re-enable cPHulk. A number of people recommended this method, but I didn\u2019t like it. I certainly don\u2019t want to disable a security measure that successfully protected the box just to be able to regain access. What would happen if a huge wave of brute force authentication attempts hit the box in the time between disabling and re-enabling cPHulk? The answer is that the box wouldn\u2019t protest and would tell the attacking program whether each attempt was successful or not.<\/p>\n
If you need to use this method, the two commands you will want to use are:\u00a0<\/span>\/usr\/local\/cpanel\/bin\/cphulk_pam_ctl --disable<\/code>\u00a0<\/span>and\u00a0<\/span>\/usr\/local\/cpanel\/bin\/cphulk_pam_ctl --enable<\/code>. These two commands will disable and enable cPHulk, respectively.<\/p>\n
I decided to use another method. This method didn\u2019t require disabling cPHulk, and thus, didn\u2019t require reducing protection to regain access. Essentially, I cleared the tables manually, so that I could log in once again. Since this opens up the login process again, a brute force could still proceed but would be quickly shut down again.<\/p>\n
While still connected to the database through the MySQL monitor, I ran a couple more queries.<\/p>\n
Note: Before I ran these queries. I stored the details of these attacks in a file on the server in case I needed to refer to the information later. Never delete attack data without keeping a record of it somewhere. You never know when you may need it.<\/em><\/p>\n
mysql><\/span> delete from brutes;<\/span>\r\nQuery OK, 0 rows affected (0.00 sec)\r\n\r\nmysql><\/span> delete from logins;<\/span>\r\nQuery OK, 32 rows affected (0.00 sec)<\/pre>\n
Now, we can log back into the box.<\/p>\n
Final Thoughts<\/h3>\n
I have to say that I was very pleased with the performance of cPHulk, a piece of software that I didn\u2019t configure and which did its job admirably with no intervention by us. I wasn\u2019t as pleased with the performance of the Liquid Web support person that \u201chelped\u201d us. Maybe I\u2019ve just been pampered with the amazing support quality that I\u2019ve experience at\u00a0<\/span>Host Gator<\/a>, but I don\u2019t think that \u201creboot the server\u201d is an appropriate response to \u201cthe security software locked us out\u201d. In fact, I\u2019m not even sure if a reboot would have done anything other than just piss us off.<\/p>\n
I sent a tweet about this yesterday and was pleased to get a response from Nick Campbell, a manager at Liquid Web, who wanted to know who recommended the reboot solution so that he could \u201cthwack them if needed\u201d. Thanks Nick.<\/p>\n
Nick gave me an update about the situation. The reboot was to connect a console since the tech couldn\u2019t access the box via SSH (due to the fact that I have disabled password auth, not due to some mysterious issue). It turns out that much of this ado was simply due to both sides not fully communicating the details.<\/div>\n
The terminal output styling is new. I worked on it last night, and I\u2019m quite pleased with the results. I think it makes the output much easier to scan and understand. I\u2019d love to know what you think about it.<\/p>\n<\/div>\n
\n
Categories :\u00a0<\/span>Tips ‘n Tricks<\/a><\/span><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Fix WHM\/cPanel cPHulk Brute Force Protection Lock Out Via SSH One of my company\u2019s servers is hosted with\u00a0Liquid Web. Yesterday one of my co-workers tries to log into\u00a0WHM\u00a0and sees the following message: Brute Force Protection This account is currently locked … Read More »<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1526","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1526","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1526"}],"version-history":[{"count":1,"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1526\/revisions"}],"predecessor-version":[{"id":1527,"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1526\/revisions\/1527"}],"wp:attachment":[{"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1526"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1526"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.wildow.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1526"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Finishing Up<\/h3>\n
I\u2019m sure that many people encountering this situation would just pat themselves on the back and go about their day, but I wasn\u2019t satisfied.<\/p>\n
I wanted to ensure that the office wouldn\u2019t be blocked in the event that another brute force attack happens, so I added our office to the whitelist. The whitelist supports both IP numbers and hostnames. So, if you have a dynamic IP, you can use a DDNS (Dynamic DNS) service to get your own hostname to use. Both\u00a0<\/span>DynDNS<\/a>\u00a0<\/span>and\u00a0<\/span>No-IP<\/a>\u00a0<\/span>offer reliable free solutions that provide you with a hostname to use. In addition,\u00a0<\/span>EveryDNS<\/a>\u00a0<\/span>is a free DNS service provider that also offers the ability to create subdomains of your personal domain that can link to dynamic IPs. Many current routers support either DynDNS, No-IP, or both so that the router can handle updating the hostname automatically. Some router firmwares, such as\u00a0<\/span>Tomato<\/a>\u00a0<\/span>support a much larger variety of DDNS services.<\/p>\n
Another nice feature about using the DDNS services and whitelisting a hostname is that you can update that DNS pointer remotely. This means that if you get locked out of WHM when you are away from the location that is whitelisted, you can update the IP on the DDNS system, wait a few minutes for the IP to update, and then log into WHM via the whitelist.<\/p>\n
One final thing I did was to change the account\u2019s password to an even more complicated and much longer one. Good luck cracking that in 32 tries.<\/p>\n