Just last month we hosted a webinar called “Heartbleed – You Stopped the Bleeding but Did You Fix the Problem?”. Heartbleed allows an attacker to retrieve the contents of memory from vulnerable servers. As a result, any private credentials that might have been resident in memory can no longer be considered private. That is why many enterprises and public facing web services advised their users to change their passwords.
What did not get much attention were other forms of credentials such as private SSH keys. One stolen private key gives an attacker access to the systems that specific private key is authorized on. For example, a web server might have an SSH private key authorized to access a back end data base. Then there is the issue of transitive trust. The data base server might have private keys authorizing access to back up systems. If a log management system is running on the same server it might have keys authorized on dozens of other servers. A couple of graphics we use to explain this issue to customers are shown below. The transitive trust graphic shows how a complex network of trust relationships can enable an attacker with just 1 stolen key to gain access across the server estate. The other graphic (we call it “The Blob of Death”) is a real world picture of trust relationships inside a modest sized data center of just a few hundred servers. There are over 8000 trust relationships in the environment of about 550 servers.
THE BLACK BLOB OF DEATH
Our first Heartbleed take away was this: Patching the vulnerability means you prevent future losses of data due to Heartbleed, but does nothing to recover from data loss you may have already experienced. You need to evaluate what might have been taken and what that information could be used for. In the case of stolen SSH private keys the potential for enterprise wide compromise is real and significant.
Our second takeaway: If you are like about 80% of enterprises and widely use SSH keys, it might not be so easy to replace potentially compromised keys with new ones. Assuming you know where are the keys are and what they are used for (most IT organizations don’t) the simple mechanics of manually changing out key pairs are daunting to say the least. It takes about 20 minutes to change a key pair (remove the public key, generate a new private/public key pair, install the keys to the proper locations, test everything works, document the change). A typical 1000 server environment will have about 15,000 SSH key based trust relationships. At 20 minutes per, we are talking 2.4 man years of work. If you want to get it done in 1 month you’ll need to allocate 29 people to the task.
The final takeaway: Even if you are able to restore your environment to a non-compromised state, what happens if there is another compromise? Do you once again either live with compromised credentials and hope for the best, or do you go through the expensive and painful process of manually replacing keys? As we said our webinar, it is only a matter of time before a new vulnerability emerges that puts SSH keys at risk. And just a few weeks later, we get Shellshock, an even worse vulnerability than Heartbleed. Shellshock enables an attacker to take control of the target system. They can not only steal private keys, they can also create new key based authorizations to use as backdoors to access the environment if the stolen keys are disabled. Heartbleed and Shellshock tell the same story: Even after you fix the vulnerability, your environment is potentially compromised. Your work is not yet done.
What’s the answer? Preparedness. Monitor and manage your Secure Shell environment so you are ready to deal with compromises if and when they happen.
Take the first step. Find out how your organization can gain visibility and control of your Secure Shell infrastructure with Universal SSH Key Manager