As many of you know, servers running SSH are under constant siege by hackers and botnets, but how are attackers getting into these servers? Servers are typically broken with brute-force password attacks because this is easy when people use passwords like "1234" and "changeMe", but do attackers do when SSH keys are used as credentials instead of passwords? We at SSH Communications Security are well aware of other attack vectors such as SSH keys. The recent SSH key scanning attacks on websites reported by Wordfence are yet another high profile example, but I like to know as much about my advisory as possible.
Wordfence has logged attacker activity as the attacker tries to scan a variety of file directory paths to find the location of any accessible private SSH keys that are not protected.
The graph below, demonstrates the sudden spike in SSH scanning traffic captured by Wordfence:
According to Mark Maunder of Wordfence, this spike in activity indicates that attackers are having success scanning the web for exposed SSH private keys and then using those keys to connect to the machines that the SSH private keys are authorized to. Attackers have found that there is an operational mistake in how users are handling SSH keys and the attackers have invested more effort in exploiting the lack of understanding and security around SSH key management.
Attacking and exploiting weaknesses in SSH management and knowledge is not new to hackers. Last year I worked on a honeypot project with Eric Wedaa of Marist College. Eric collected a lot of wonderful stats on the type of attacks his sacrificial SSH servers (honeypots) sustained from around the world. Attackers use machines and programs called botnets to scan the internet for unprotected machines. Eric told me that it used to take days for one of his honeypot to be attacked, but now it seems to take hours or minutes for an attacker to find and attack the honeypot. Clearly attackers are having success and getting more aggressive.
A honeypot is a server that looks legitimate to attackers, but in reality the purpose of this server (honeypot) is to collect information on attack patterns by hackers. This information helps us better understand what methods hackers are using to attack servers. In the case of the Marist honeypot, called LongTail, an OpenSSH server was modified to log information about anyone who attacks this server. The honeypot saves information such as the username, password, and IP address the attacker uses to try and get into the server. The trick with honeypots is to make the server look just like a normal server. If the attackers notice anything unusual about the server, they will suspect it is a trap or honeypot and move to the next target. The Marist LongTail honeypot fools attackers by using the real OpenSSH code with small modifications and running this code as the SSH server on the honeypot server. The honeypot looks just like a real OpenSSH server because it is using the actual OpenSSH code.
Most of the attacks on the Marist honeypot are SSH brute-force, meaning that the attacker just guessed common user passwords until they were successful. The vast majority of attacks came from China. One key reason for this is cybercriminal groups similar to the China based SSHPsychos. This group is so active in their brute-force attacks that at times they account for up to 35% of all SSH traffic on the internet.
Passwords are susceptible to brute-force attacks for a number of reasons, but one of the main reasons is that people use insecure or easily guessed passwords. The LongTail project proves this by collecting the most common passwords attempted by attackers. The most common passwords attempted on the root (privileged admin account) are about as bad as you would expect, but with a few surprises: root, wubao, admin, 123456, password, 1234, and jiamima. Wubao translates from Chinese to ‘without protection’ or ‘no password’ and jiamima translates to ‘encryption key’ or ‘password’. As mentioned before, attackers are attempting these methods because they work on some people.
See the most 20 common root passwords bar chart for LongTail here:
Clearly SSH is under constant password brute-force attack, but Eric felt like this was just the tip of the iceberg. Eric suspected that attackers must be trying to exploit SSH keys in the same way. These could be SSH private keys that were duplicated and propagated throughout an environment or SSH private keys that were mistakenly exposed to the network as the Wordfence report details. After hearing Eric’s story, I agreed to help him collect data on SSH key attacks. So I made some updates to the Marist LongTail OpenSSH code to capture SSH key data when an attacker attempts to breach SSH using a compromised key.
At the time the one honeypot we deployed the updated code to capture SSH keys showed no attacks. This was probably due to the fact that it was only deployed on one honeypot and deemed a low value target by the attackers. In other words, a university virtual machine is not worth targeting with a more advanced SSH key attack when brute force password attacks are working and the return on investment for the attacker is low.
This was about a year ago and things have really changed since then. Thanks to Wordfence, we now have evidence attackers are scanning for SSH keys to use. The reason Wordfence has spotted nefarious SSH key activity and LongTail has not (yet) makes a lot of sense because the target webservers are much higher value targets than our Marist honeypots were and there are many more webservers to pull data from than the number of honeypots Marist LongTail has setup.
SSH keys are credentials, just like passwords, but there are a lot of complicated differences between SSH keys and other credentials such as passwords and certificates. Understanding these complex differences is paramount to properly manage and secure SSH key access.
An SSH key pair contains both a private and public key. The private key is stored on the machine you are connecting from and the public key is kept on the remote machine you are connecting to. SSH private keys, as the name implies, should never be shared or exposed to the wider network because this key allows anyone to impersonate the owner of the private key and to connect to any machine that private key is authorized to connect to. You should never put them where other people can read them, send them in an email, FTP them, or transmit them insecurely.
To properly control your SSH key access you first need to discover what SSH keys you have, what machines they can access, and determine what SSH keys violate compliance regulations by being too old or too weak. From there it is important to monitor SSH key activity over time to determine what SSH keys are unused or obsolete. Unused keys should be removed to reduce your attack surface and to gain quick wins. Some companies have more than a million unused SSH keys waiting to be exploited by attackers.
To prevent a small breach from turning into a headline grabbing disaster, we recommend you assess how far an attack can spread across your enterprise with an SSH Key Risk Assessment.