Episode 232: Log4j Won’t Go Away (And What To Do About It.)

In this episode of the podcast (#232), Tomislav Peričin of the firm ReversingLabs joins us to talk about Log4Shell, the vulnerability in the ubiquitous Log4j Apache library. Tomislav tells us why issues related to Log4j won’t be going away anytime soon and how organizations must adapt to deal with the risk it poses.

---

If you’ve been paying attention to your infosec news feed this week, you’ve been inundated with stories and headlines about something called “log4j, a (previously) obscure library that is a common component of a number of Apache software frameworks. This quiet little soldier of the open source software world,  we now know, has a glaring security hole in it that allows remote code execution on affected systems. 

Episode 218: Denial of Sustenance Attacks -The Cyber Risk To Agriculture

Log4j: A Very Popular Library

And that’s a big problem. Why? Well, it turns out that Log4j is a very, very, very popular software library. The firm Sonatype notes that in November, log4j-core, the vulnerable version of the module, was the 252nd most popular component by download volume in Sonatype’s Maven Central code repository. That’s out of a total population of 7.1 million artifacts – that’s the top 0.003% percentile in popularity by downloads. To date, more than 2000 software packages have been identified that are potentially vulnerable to attacks targeting log4j. Those include both the popular Minecraft massively multiplayer online game as well as Apple’s iCloud and Twitter. SAP announced on Wednesday that it, alone, patched 20 applications that used Log4j. In the meantime, threat actors are scanning the Internet to identify servers vulnerable to exploitation.

Episode 208: Getting Serious about Hardware Supply Chains with Goldman Sachs’ Michael Mattioli

Supply Chain Risks: The New Normal

What does this mean for your organization? And what does the Log4j vulnerability tell us about the shape of cyber risks and threats to come? We invited Tomislav Peričin in to the Security Ledger studios to talk. Tomislav is the Chief Software Architect at the firm ReversingLabs and he’s an expert in software analysis and supply chain risks. In this conversation, Tomislav explains what Log4j is and why the security hole in it poses such a big risk. He also talks about some of the big picture changes that organizations will need to make to stay on top of supply chain risks such as this. At the top of that list of changes: creating and analyzing software “bills of materials”(SBoMs) that allow you to keep track of the ingredients in the applications  you rely on. 

---

Episode 232 Transcript

[START OF RECORDING]

PAUL: Hello and welcome to the Security Ledger podcast. I’m your host, Paul Roberts, Editor In Chief at the Security Ledger. In this week’s episode of the podcast, number 232…

TOMISLAV: I think we will be patching this for months because the scope is still unknown. I think a lot of companies, open-source projects, they just responded immediately and started to update things. But the problem with transient dependencies is that you might not even be aware. So it’s going to be a lot of patching for a lot of folks. And I think people are going to be abusing this vulnerability for the foreseeable future.

PAUL: If you’ve been paying attention to your InfoSec newsfeed this week, you’ve probably been inundated with stories and headlines about something called Log4j, a previously obscure open-source library that’s a common component of a number of Apache software frameworks. This quiet little soldier of the open-source software world we now know has a glaring security hole in it that allows remote code execution on systems that use the library. And that’s a big problem. Why? Well, as it turns out, Log4j is a very, very, very popular software library. The firm Sonatype notes that in November alone, Log4j core, the vulnerable version of the library, was the 252nd most popular component by download volume in the company’s Maven central code repository. That’s 252nd out of a total population of 7 million software artifacts. To date, more than 2000 software packages have been identified that are partially vulnerable to attacks targeting Log4j. Those include both the popular Minecraft massively multiplayer online game as well as Apple’s iCloud and Twitter. SAP alone announced patches for 20 applications that it discovered using the vulnerable version of Log4j. And in the meantime, threat actors across the globe are scanning the Internet to identify servers and other applications vulnerable to exploitation. Among those are both nation-state and cybercriminal groups. What does this mean for your organization and what does the Log4j vulnerability tell us about the shape of cyber risks and threats to come? In this week’s episode of the podcast, we invited Tomislav Peričin, the Chief Software Architect and cofounder of the firm ReversingLabs, into the studio. Tomislav is an expert in software analysis and supply chain risks, and in this conversation, Tomislav explains what Log4j is and why the security hole in it poses such a big risk. He and I talk about some of the big picture changes that organizations are going to need to make in the months and years ahead. At the top of the list of his recommendations, software bills of material that allow organizations to keep track of the various ingredients in the software and services that they rely on. To start off, I asked Tomislav to introduce himself and tell us a little bit about the company he cofounded, ReversingLabs.

TOMISLAV: My name is Tomislav Peričin. I’m the Chief Software Architect and one of the founders of ReversingLabs.

PAUL: Tomislav, welcome to Security Ledger podcast.

TOMISLAV: Thank you.

PAUL: It’s great to have you.

TOMISLAV: Good to be here.

PAUL: We’re here talking to you. It’s Wednesday, and we’re not even a week out from the first report of this very serious vulnerability. Log4j, not a particularly memorable name for vulnerability, but it is the big security story this week and one that actually seems to kind of live up to the hype in terms of its impact. So I thought we’d have you on just to update our listenership on what this thing is and why they should be concerned about it. Before we do that, though, could you just talk a little bit about ReversingLabs and what ReversingLabs does and the work you do there?

TOMISLAV: Well, absolutely. I mean, ReversingLabs is a security company. We’ve been in business for more than a decade now. We started in 2009. Our mission statement at the time was to be able to analyze any kind of binary payload and basically analyze files to static analysis. Decompose them as best as we can collect information about files and do classification. It all grew way, way more than just that initial mission statement. But now we also analyze software and look for things like vulnerabilities, software composition analysis, various aspects of software quality in general. And I guess today we are going to talk about a little bit of that, which is referring to the Log4j vulnerability.

PAUL: So let’s talk about Log4j. I guess a good place to start is what is Log4j? It’s a component of the Apache software, but what does it do exactly? What’s its purpose?

TOMISLAV: Yeah. Log4j is just a typical Java library. There’s many different libraries of the similar nature. This one, in particular, is basically used for logging purposes. So when you’re developing an application, there’s a need to log different events, essentially to either like command output or textual files, or even log them to a remote server. That’s exactly what this library does. It enables the Java programmers to be able to rapidly implement a robust logging system into their application, and it’s basically plug and play for most Java developers. There are other libraries of similar nature which kind of use similar interfaces, and they’re going to be called in a similar fashion. Do not get confused by that. This is a vulnerability itself is referring to the original library called Log4j.

PAUL: And this is part of, there are a number of different Apache frameworks that Log4j is a part of. Right?

TOMISLAV: Yeah absolutely. So Log4j is a very popular library, and there’s a huge proliferation in this segment, meaning that there’s a lot of applications out there which are using Log4j as one of its dependencies. I don’t know the number by heart, but it’s definitely in tens of thousands.

PAUL: Sonatype had some really interesting data on downloads of Log4j and even just the vulnerable versions, we’re in the top 250 of 7 million software components that they track. So obviously very widely used. If you’re a software application developer, you might not be making a discrete decision to use Log4j. Right? It would be part of a framework that you chose to use, but made that decision kind of at a much higher level. And Log4j was just part of that framework, right?

TOMISLAV: Yeah. The problems with developing application nowadays, and I said problems loosely there, is that they’re based on a large number of third party dependencies, and as this ecosystem grows, not all dependencies are self contained. They tend to also depend on other dependencies within the same ecosystem. And tracking the dependencies of your dependencies is a very difficult thing to do. So developers out there could be using Log4j, something referred to as transient dependency. So dependency if your dependency meaning that you might not even be aware that Log4j and especially the vulnerable version, is being used in your code base.

PAUL: Yeah. Because it’s not only about tracking the components, it’s also tracking which version of the component is used. Right? So it’s an even harder task than just knowing what’s in the frameworks you’re using, it’s what version of what’s in it?

TOMISLAV: Correct. And that’s exactly the nuance point here that needs to be made is that there are multiple versions of Log4j library. There’s one X branch, if you will, and two X branch. And this particular vulnerability that everybody’s talking about lot for Shell is only affecting two point X branch, and only a sub segment of those versions are actually known to be vulnerable. Luckily for us, there’s already a patch. The maintainers have issued actually two updates to the library, so 2.15 and 2.16, both of them addressing the issue. And another vulnerability, which was just discovered as pretty much every developer and researcher is looking at this library at this point in time.

PAUL: Let’s talk about the vulnerability itself. It’s rated critical. It’s a remote code execution vulnerability, meaning that you don’t need to be a local user or admin, and it’s considered a fairly low sophistication or low skill level exploit to leverage. Talk about how this vulnerability would be exploited by a malicious actor if they found a vulnerable application.

TOMISLAV: Well, first on the criticality, right. Just so listeners are aware, as we kind of grade the severity of these issues, there’s a scale from one to ten, ten being the worst one. This vulnerability has been graded as ten out of ten, meaning that’s the most severe vulnerability that you can possibly have. So that’s why it takes the most attention. And that’s why it’s making all the news as well. So that’s in terms of criticality. In terms of what does it actually do? How does it affect you? It’s not that difficult to explain really. Since Log4j is a logging library, it can log any textual string an application needs to log. If that string is controlled by the user that’s being logged, so any arbitrary string. Then that string can be parsed by the library, and that string could cause the library to load the remote resource. So quite literally, to ping back from wherever the login library is running to a remote server, expose the environment information where the library is running, but also load a remote class into the context of that login library and execute that code. So that’s why we’re saying it’s a remote code execution vulnerability, because just by sending out a string to a server, if that string is being logged through this library, it can cause remote code execution.

PAUL: And it should be noted, the researcher who discovered this and reported it was based in China, if I’m not mistaken and worked for Alibaba, is that right? Did I get that right?

TOMISLAV: That’s as much as I gathered as well. I mean, it seems like this conflicting stories really has been discovered in the gaming community. And then the Alibaba guys actually started to document the issue, and then it just started to proliferate through the Internet.

PAUL: And one of the platforms we’ve heard the most about in terms of exploitation is Minecraft. Is that right?

TOMISLAV: Yeah. It sounds weird, but yeah, Minecraft is one of those Java based applications that does use Log4j for login purposes, and they are logging arbitrary user strings, and therefore you could inject remote code execution this way. I guess somebody really wanted to win at Minecraft.

PAUL: Leave it to the gamers. They’re a determined bunch, but there are others as well. And in fact, iCloud is affected. Twitter is affected. There’s a list of something like 1600 or more platforms and applications that are vulnerable to this, and it is a pretty cultivated list.

TOMISLAV: Yeah, it’s growing, growing by the minute. I think various organizations like C-CERT and CISA as well. They’re all been trying to compile the list of vulnerable and non vulnerable applications and try to coordinate this massive patching wave. That’s about to hit us. There’s going to be a whole bunch of applications just issuing patch releases. And it’s going to be like this for a while because it’s going to take a lot of time for people to figure out if they’re actually dependent on this life.

PAUL: Yeah, so let’s talk about that. How do you figure out if you’re exposed to this vulnerability? Again, given how widely used it is not only by different Apache frameworks, but also, of course, by third party applications that then leverage those frameworks and that you might be a consumer of yourself.

TOMISLAV: Right now, that’s a million dollar question. That’s the question everybody’s asking themselves. And honestly, we have had a whole bunch of support tickets as well. Our customers reaching out to us literally asking, are you affected by this vulnerability? Do you have a public statement about if you are, what are the effective products and the rest of it? And we had to take as everybody else we had to take a really deep look into our entire code stack and figure out where are these points where we can even have Log4j as a transient dependency. And answering that question is difficult because not everybody is right now publishing their software build materials, which is pretty much the essential way of knowing about dependencies in software packages. Doing that is likely going to become mandatory in the near future just because of everything which has happened during last this year, we have solar winds attack, which started this discussion. And we had the executive order signed by President Biden about looking into making software bill materials mandatory, at least in government use. And now we have even a minimum set of requirements defined for how software bill materials should look like. But its key value is the ability to create the software inventory. So when an attack like this or vulnerability like this actually happens, you have a place where you can ask this question and you can get an answer of where is it located? What do I need to update? What do I need to take offline? All those questions become a lot easier to answer than they are now.

PAUL: So did ReversingLabs do that for your own software?

TOMISLAV: Absolutely. We have actually built a platform that creates software billing materials for third party software, even if that’s not provided with the whole notion of automatically creating this software inventory. So then when this type of an event happens, we have a place to ask that question. And as soon as the news broke, we were like, probably like in 15 minutes, knew all the affected services. What do we need to patch? What’s the exposure, even if we couldn’t update, and there were a few services like that immediately to at least isolate and apply the proposed workarounds by the maintainers.

PAUL: So this brings up, as you said, this whole issue of software billing materials, and it’s a term that the Biden administration and CISA and others have started to throw around. And folks have been talking about this concept for a long time, but my sense is out there in the bigger world of software publishing, application development, let alone the enterprise space, this is still a pretty new concept. So just to make clear, software billing materials is something both for software publishers and for their customers, right, to be able to understand what’s in the sausage that we’re making? And also, I guess the sausage that as a company we may be consuming.

TOMISLAV: Absolutely. We provide it for our own software packages, so our customers know exactly what we’re depending upon, and they could then create these software inventories for themselves if they choose to do that. And we’re also generating these software materials for third party software components that don’t provide it just yet. So even if this becomes a requirement, not everybody will be able to do it, not do it correctly, not keeping it up to date. All of those things are at the crux of why S Bomb is so critical and these guys, the maintainers are this is an open source package, really, there’s only so much time in the day for them to do everything and asking them to provide materials and all the other things that we actually depend upon. It is a bit of a stretch, so we will need to have these systems that can automatically generate and inspect and validate software materials moving forward.

PAUL: So if you have one of these, let’s say you’re buying software as an enterprise or you’re producing software and you’re providing this to your partners and your customers… Does it just sort of, if you were to look at it, does it just sort of list out all the various software components that are in the recipe of this application that you’re selling? Or how does it, what does the software bill of materials look like? And how can it be operationalized?

TOMISLAV: Well, that’s the general idea. Typically, it does the best effort to do that, even though it doesn’t necessarily always list out all of the components. There are certain things which are not known to the producers of S Bomb because you have to take into consideration a lot of these software materials are being manually created. Therefore, they’re not going to be completely accurate. And even if they are programmatically generated, there are certain libraries which are niche or smaller in scope and therefore wouldn’t appear. Exactly, Log4j definitely would appear on most software bill of materials, and you would definitely know that it’s there. I’ll just tell you how our CISO actually uses it. Literally collect all the S Bombs, put them in a single place and monitor them for vulnerabilities. And as long as something like this happens, then we have the central depository. We can ask questions and get exposure information. There are open source tools out there that work with the standard S Bomb formats like Cycle DX, and they enable you to create this graph of dependency and ask it questions. So that’s pretty handy, but you’re still dependent on the information and the accuracy of information in the S Bomb. So that’s why generating the S Bomb is going to become so critical.

PAUL: How do you, if you’re vulnerable to this, how do you get out from under it? What is the process for not being vulnerable to the Log4j vulnerability?

TOMISLAV: Well, first, I think it’s unfair to call it a vulnerability. Certainly it is a security issue. No one’s debating that. I just think that this was added as a function that was requested and as such should be taught as a feature. And from the security standpoint, I think it’s just a case of unsafe default, default configuration. It shouldn’t be enabled by default, and that’s exactly where the maintainers have what they’ve done right now. As one of the mitigations in the latest patch, this is now disabled by default and you can enable it should you need it. And it should have been like that from the get-go. That doesn’t necessarily matter at this stage. These things happen. But I think adding security review for these kinds of feature modification requests is a good idea. Now, probably the project didn’t have security support at that stage. They were tiny or underfunded, and all the other things all that can happen just moving forward. I think future modification requests like this should be put under scrutiny just because we’ve just now learned something very important.

PAUL: And this is like a big issue, I think just generally now, especially as we’re becoming more and more reliant on open-source components. Like many of these libraries and components that are very, very widely used, might be created and managed by a single developer or a very small team of developers. They were developed to do a very specific thing. But as you pointed out, might not have had the most robust security model or testing in place to anticipate all of these potential misuses. But then they end up used all over the place, including in very sensitive applications. And it’s just sort of like a gap that exists in the current system. It seems like to me, right?

TOMISLAV: Oh, absolutely. I think always think back to that XKCD sketch, which kind of has these blocks on top, and there’s this one guy in Nebraska maintaining this library unpaid and so on for a very long time. In reality, it is a lot like that. And what I think is S Bomb is going to help us actually do is identify the critical stuff. Right? Like, if you had the S Bomb for all of these packages that you’re using in your organization, let’s start there. Right? You would know that this is a highly used library and is proliferated across all of your code base. Therefore, having that single point of failure makes it a high risk. So I think we’re going to start as an industry to think like that, and that might lead to better security budgets for these components. That would be ideal. But it also might lead to kind of fragmentation in the libraries themselves, as people kind of start to mitigate their risks by using more.

PAUL: But that’s okay, right?

TOMISLAV: Yeah. That’s one strategy.

PAUL: So final question. If you are concerned about this as an organization or at least concerned about a tax against this, are there ways that you can monitor things you should be looking for on your network to spot potential attempts to exploit this?

TOMISLAV: That’s a very complicated question to be honest.

PAUL: That’s why I saved it for last.

TOMISLAV: Now, people are scrambling. So there are web application firewall rules that people have created. They’ve been embedded in the products themselves. But the problem is this protocol can be obfuscated in almost unlimited number of ways. So people are creating rules right now, and they’re being constantly updated, but they’re just trying to match what’s actually being used in the wild and whack them all. Yes. Exactly. I don’t think we can have a good generic role at this stage. But looking at the patterns of that LDap and J and the I strings in the logs would be a good start.

PAUL: And what’s your expectation about what we’re going to see with this vulnerability in terms of the impact it’s going to have just in the threat landscape?

TOMISLAV: I think we will be patching this for months because the scope is still unknown. I think a lot of companies did really good and open-source projects. They just responded immediately and started to update things. But the problem with transient independencies is that you might not even be aware. And without the bomb, it’s really going to be hard to figure all that out. So it’s going to be a lot of patching for a lot of folks. And I think people are going to be abusing this vulnerability for the foreseeable future.

PAUL: Tomislav Peričin, thank you so much for coming on and speaking to us.

TOMISLAV: Thank you very much for having me.

PAUL: It was a pleasure. We’ll do it again… Tomislav Peričin, the Chief Software Architect at ReversingLabs. He was here in the studio to talk to us about the Log4j vulnerability.