IPMI’s Inconvenient Truth: A Conversation With Dan Farmer

The work of brilliant computer security researchers often borders on a kind of madness. After all, it takes dedication and a certain amount of monomania to dig through the mush of disassembled source code or the output of application fuzzers and find the one software vulnerabilities – or chain of vulnerabilities – that might lead to a successful attack.

Dan_Farmer_Thumbnail
Farmer is warning of the danger posed by insecure implementations of IPMI, which is used to remotely manage servers in large-scale deployments.

Often, this work puts you at odds with what most of us consider “the real world.” Notably: the well-respected researcher Dragos Ruiu had many in the security community wondering about his sanity after he sounded the alarm about a super stealthy piece of BIOS malware he dubbed “BadBIOS” that seemed to be everywhere and nowhere, all at once.

Dan Farmer finds himself in a similar position as he continues to sound alarms about the security threat posed by insecure implementations of the Intelligent Platform Management Interface (IPMI)– a ubiquitous protocol used to do remote management of servers.  As we wrote earlier in the week, Farmer has released a new paper that claims hundreds of thousands of systems, globally, are vulnerable to attacks on what Farmer characterized as “basic configuration and protocol weaknesses.” Those attacks could give those malicious actors total control over vulnerable servers on corporate networks or in cloud deployments and would be almost impossible to detect and stop, Farmer wrote.


Get the New 2017 SANS Research Report on 'Threat Hunting' -- Written by experts from the SANS Institute, the survey reveals a number of interesting data points about the challenges and benefits of threat hunting.


Not that anyone is questioning Dan’s sanity. But his latest report, like a similar one released last year, have been met with stony silence from firms like Intel, IBM, HP and Dell, which support IPMI and distribute their own versions of it in hardware they sell to corporations and other large organizations globally. That has Farmer wondering whether the lack of response is because he’s talking about ‘inconvenient truths’ (my choice of words, not his) or chasing shadows.

“I honestly think that you’d be crazy not to abuse this stuff if you were interested in any sort of espionage, attack, warfare, or whatever,” Farmer wrote. “I’ll reiterate that it could be that …no one really believes me that the problem is as bad as they think, or that I’m simply wrong: not only dead wrong, but I’m simply too blind/stubborn/etc. to admit it.”

To help understand the IPMI issue a bit more and parse Dan’s latest paper, “Sold Down The River,” he and I did an e-mail interview this week. Here’s how it went down:

Farmer's report, "Sold Down the River" suggests that hundreds of thousands of systems on the public Internet are vulnerable to IPMI-based attacks.
Farmer’s report, “Sold Down the River” suggests that hundreds of thousands of systems on the public Internet are vulnerable to IPMI-based attacks.

Security Ledger: In general, I see the latest paper as just putting some numbers around the problems you first talked about with your “Freight Train to Hell” report last year. Specifically: there are major problems with both IPMI Versions 1.5 and 2.0 especially in regard to a lack of authentication and/or weak authentication. Is that right, or am I missing something important? 

Dan Farmer: You are 100% spot on. I did some light scanning last year, but decided to do some more substantial efforts this year because (a) HD (Moor) had done the initial data run already and (b) I was off to the Pentagon (part of DARPA’s CFT effort) and was interested what they really were.

I should have figured this out last year; I had similar numbers, but I simply ran out of time to do anything but fairly superficial analysis on the data. (To be fair I’m not getting paid for this stuff!) But I didn’t do a very basic transformation that would have showed me a much better picture. As an example of this, I might have seen something like:

Roughly 50% of all systems that spoke the IPMI protocol are vulnerable to problem XXX, and roughly 50% of all systems are vulnerable to problem YYY

That sounds bad, but actually there was almost no overlap to the to two problems; if I’d have noticed that nearly all the systems vulnerable to problem XXX spoke one of the two versions of IPMI (either 1.5 or 2.0), and almost none of other were, and vice-versa for problem YYY, it means that instead of 50% overall with problems, you have nearly 100% of one version are vulnerable to XXX, and nearly 100% of the 2nd version are vulnerable to YYY, the conclusion is dramatically different – that nearly all systems were vulnerable.

But I also found it interesting in that there still are almost no changes or tools out there that can help folks out, a year later. I don’t suppose there will be widespread change in the next either, but time will tell. 

Security Ledger: You say that of the problem is already widespread: 230,000 Internet connected systems and an untold number of systems that aren’t directly identifiable from the public Internet. Why have we not seen more IPMI based hacks, then? Or are these attacks that may well go unnoticed?

Dan Farmer: I’ll start by saying that I don’t know the real answer here.

I should also say that I’m probably near – or at – the top of the list saying how bad the problem is. Most simply say things like “meh, no news”, or “no big deal, just keep this stuff hidden”, or “heck, only 250k servers, who cares, much larger fish to fry”, with a few “wow, this is bad, but not *too* bad.”

Perhaps obviously I think they’re mistaken, but they probably think the same of me, and there’s a hell of a lot more of them than I, so… as Groucho is apocryphally said to have said, “Who are you going to believe, me or your lying eyes?”

I do think to fully understand the problem you need knowledge of three rather complex issues:

  • how IPMI/BMCs are used in rollouts & large scale organizations
  • the protocol and capabilities of IPMI and BMC
  • the security problems/facets that are inherent or nearly so in the underlying architecture of how IPMI is used.

Without all three, you might have some difficulties swallowing what I’m saying. With all three, I may have delusions of grandeur. I’ve been very reluctant to write proof-o-pudding code – exploits or whatnot that more clearly demonstrates some issues – because I feel that such things tend to cause trouble with no clear benefit, but perhaps that’s the next step to illustrate the problem.

In any case, I’ve yet to hear any substantive critiques why any conclusions I’ve made are erroneous.

Security Ledger: You say the impact of this could be especially tough in organizations that are conducting large-scale rollouts and provisioning, and using IPMI for the management and deployment of racked servers, including cloud infrastructure. Explain?

Dan Farmer: The problem I was trying to discuss was with large-scale rollouts (in particular.) Some cloud deployments fall into that – you have a bunch of servers that either provide internal, external, or customer facing systems that implement the cloud. Modern clouds (as typified by EC2 services/servers), however, generally don’t expose hardware to the end-user – instead they rely heavily on virtual machines or services.

What this means is that the cloud *infrastructure* is troublesome from a managed interface or network aspect, but those interfaces or management points aren’t generally directly accessible from the outside (although if they are, that’s serious trouble!) The virtual systems don’t generally have any sort of access to the underlying hardware or management components any more than any other semi-random system.

But there’s another bit, as well. It has been shown in a refereed conference paper that the BMC and undocumented IPMI calls have granted full memory access to its server. This means that compromises from these systems can essentially invisibly get data or services that users consume in the cloud. But in general if you can compromise a server, you can compromise it’s BMC, and most definitively vice-versa.

And there’s more – while I’ve yet to see any sign of any widespread issues because of all this stuff, that doesn’t mean it’s not going on (or the converse.) But if you were to ask *me* to do some serious compromising of large numbers of important servers or organizations, there’s no question I’d hit out-of-band communication networks and devices as soon as possible.

Of course, there are zero tools to detect someone doing something from down under (BMC/IPMI), although they’re just starting to show in the vulnerability scanners, IDS, and IPS… they’ve really just scratched the surface. And don’t get me started on DMTF, CIM, and other management protocols that rest on top of IPMI and the BMC. They’re designed to further the ease of large scale out-of-band management and are amplifying the power of the BMC.

At the least: if you thought Stuxnet et al were stealthy, Stuxnet may as well be shooting off fireworks and cannons to announce its presence vs. what you could do with the potential stealth of compromised BMC (or groups of the same.) Night and day difference – you simply cannot detect the BMC’s actions or behavior without compromising the BMC yourself.

I honestly think that you’d be crazy not to abuse this stuff if you were interested in any sort of espionage, attack, warfare, or whatever. I’ll reiterate that it could be that either they’re off doing it behind closed rooms, that no one really believes me that the problem is as bad as they think, or that I’m simply wrong: not only dead wrong, but I’m simply too blind/stubborn/etc. to admit it.

Security Ledger: You say that organizations do not typically have the tools needed to monitor or manage IPMI/BMC security. What steps are available to organizations that at least want to survey their infrastructure and assess their risk?

Dan Farmer: Well, I wrote up a small summary in the paper; I also cite my best practices, and I’ve written some small tools to help, but other than the best practices they’re more ad hoc. Some small inroads are being made – I’ve seen the Metasploit and Nessus scanners start to uncover this, but it’s still really in its infancy.

Security Ledger: Have you had any contact with the vendors you mention: Intel, IBM, Dell, HP, SM? Do you have any sense whether there will be efforts to address some of the issues you raise in this report and Freight Train?

Dan Farmer: Almost none in an official capacity, and zero who actually want me to say anything about our communications.

I’ve gotten some very interesting conversations from folks out-of-band (so to speak ;)) that work for them. Most don’t concur, but on the other hand they also don’t try to convince me of their reasons. I’ve only talked to people at Intel, Dell, and SM, and all at my request, although I’ve had some out-of-band communications with some others.

Recently a person from one of the companies that was a charter member in the original IPMI specification told me that ‘management interfaces were never meant to be placed on the Internet.’ Well, three things in response to that:

  1. perhaps, but the spec never says that
  1. vendors give zero guidance about this stuff in or out of the spec, and
  1. if you have a server on the Internet – web, mail, whatever – then it’s management interface is on the internet, since if you break into the server you break into the management interface.

They didn’t reply to me after that.

I talked to another monster server company last year; turns out they had code that looked like QA or test code in their BMC product that allowed easy access to the deep underpinnings of their BMC, including shell access, etc. I told them, they scrambled to fix it, but when I asked them how this code got there, they said “we have no idea.”

That said, they pulled all the firmware including some historical versions (apparently a first for them, as they keep all the old firmware on their website) (but) didn’t mention the problem, and quietly moved on. Was it their folks? Did someone put it there? Did someone break in and leave it there? Was it some foreign power? Who knows.