In this episode of the podcast (#200), sponsored by Digicert: John Jackson, founder of the group Sakura Samurai talks to us about his quest to make hacking groups cool again. Also: we talk with Avesta Hojjati of the firm Digicert about the challenge of managing a growing population of digital certificates and how automation may be an answer.
Life for independent security researchers has changed a lot in the last 30 years. The modern information security industry grew out of pioneering work by groups like Boston-based L0pht Heavy Industries and the Cult of the Dead Cow, which began in Lubbock, Texas.
After operating for years in the shadows of the software industry and in legal limbo, by the turn of the millennium hackers were coming out of the shadows. And by the end of the first decade of the 21st century, they were free to pursue full fledged careers as bug hunters, with some earning hundreds of thousands of dollars a year through bug bounty programs that have proliferated in the last decade.
Despite that, a stigma still hangs over “hacking” in the mind of the public, law enforcement and policy makers. And, despite the growth of bug bounty programs, red teaming and other “hacking for hire” activities, plenty of blurry lines still separate legal security research from illegal hacking.
Hacks Both Daring…and Legal
Still, the need for innovative and ethical security work in the public interest has never been greater. The Solar Winds hack exposed the ways in which even sophisticated firms like Microsoft and Google are vulnerable to compromised software supply chain attacks. Consider also the tsunami of “smart” Internet connected devices like cameras, television sets and appliances are working their way into homes and workplaces by the millions.
What does a 21st century hacking crew look like? Our first guest this week is trying to find out. John Jackson (@johnjhacking) is an independent security researcher and the co-founder of a new hacking group, Sakura Samurai, which includes a diverse array of security pros ranging from a 15 year old Australian teen to Aubrey Cottle, aka @kirtaner, the founder of the group Anonymous. Their goal: to energize the world of ethical hacking with daring and attention getting discoveries that stay on the right side of the double yellow line.
In this interview, John and I talk about his recent research including vulnerabilities he helped discover in smart television sets by the Chinese firm TCL, the open source security module Private IP and the United Nations.
Can PKI Automation Head Off Chaos?
One of the lesser reported sub plots in the recent Solar Winds hack is the use of stolen or compromised digital certificates to facilitate compromises of victim networks and accounts. Stolen certificates played a part in the recent hack of Mimecast, as well as in an attack on employees of a prominent think tank, according to reporting by Reuters and others.
How is it that compromised digital certificates are falling into the hands of nation state actors? One reason may be that companies are managing more digital certificates than ever, but using old systems and processes to do so. The result: it is becoming easier and easier for expired or compromised certificates to fly under the radar.
Our final guest this week, Avesta Hojjati, the Head of R&D at DigiCert, Inc. thinks we’ve only seen the beginning of this problem. As more and more connected “things” begin to populate our homes and workplaces, certificate management is going to become a critical task – one that few consumers are prepared to handle.
What’s the solution? Hojjati thinks more and better use of automation is a good place to start. In this conversation, Avesta and I talk about how digital transformation and the growth of the Internet of Things are raising the stakes for proper certificate management and why companies need to be thinking hard about how to scale their current certificate management processes to meet the challenges of the next decade.
(*) Disclosure: This podcast was sponsored by Digicert. For more information on how Security Ledger works with its sponsors and sponsored content on Security Ledger, check out our About Security Ledger page on sponsorships and sponsor relations.
As always, you can check our full conversation in our latest Security Ledger podcast at Blubrry. You can also listen to it on iTunes and check us out on SoundCloud, Stitcher, Radio Public and more. Also: if you enjoy this podcast, consider signing up to receive it in your email. Just point your web browser to securityledger.com/subscribe to get notified whenever a new podcast is posted.
Episode 200 Transcript
[START OF RECORDING]
PAUL: This week’s Security Ledger Podcast is sponsored by DigiCert. DigiCert is the world’s premier high-assurance digital certificate provider simplifying SSL, TLS, and PKI and providing identity, authentication, and encryption solutions for the web and the Internet of Things. Check them out at digicert.com.
Hello, and welcome to The Security Ledger Podcast. I’m Paul Roberts, Editor in Chief at The Security Ledger. In this week’s episode, number 200…
AVESTA: Transformation doesn’t happen overnight but what could potentially happen overnight is a big catastrophe where your infrastructure could be down because you did not spend the days upgrading and changing and having a very solid infrastructure.
PAUL: The covid pandemic has slashed man for a wide range of products from gasoline to suits and dress shoes but business in digital certificates to secure online transactions and other activity is booming, pandemic or not. Alas, more certificates means more work for companies who hope to manage those certificates and keep them safe and secure. In our second segment this week, we’re joined by Avesta Hojjati of the firm DigiCert to talk about the challenge of managing a growing population of digital certificates and how automation may be the answer for over-stressed IT groups. But first, life for independent security researchers has changed a lot in the last thirty years. Modern information security industry grew out of the pioneering works of groups like L0pht Heavy Industries in Boston and the Cult of the Dead Cow. After spending decades operating in the shadows of the software industry and in legal limbo, vulnerability hunters now have full-fledged careers earning hundreds of thousands of dollars a year through bug bounty programs.
But a stigma can still hang over security researchers and despite the emergence of bug bounty programs, there are still plenty of blurry lines separating legal security research from illegal hacking. Despite that, the need for innovative and ethical security work has never been greater. What does a 21st century hacking crew look like? Our next guest is trying to answer that question. John Jackson is an independent security researcher and the co-founder of a new group, Sakura Samurai, which includes a diverse array of security pros ranging from a fifteen-year-old Australian teenager to Aubrey Cottle, AKA Kirtaner, the founder of the group Anonymous. Their goal is to energize the world of ethical hacking with daring and attention-grabbing discoveries that stay on the right side of the double yellow line. In this interview, John and I talk about his recent research including vulnerabilities he helped discover in smart television sets by the Chinese firm TCL, a common open-source security module used worldwide, and infrastructure owned and operated by the United Nations. To start out, I asked John to talk a little bit about himself and about Sakura Samurai, the group he founded.
JOHN: So, I’m John Jackson and I’m an application security engineer and the founder of Sakura Samurai.
PAUL: John, welcome to The Security Ledger Podcast.
JOHN: Thanks for having me.
PAUL: It’s great to have you. You and I have actually been talking for a few months now about a bunch of different stories that Security Ledger has written on TCL Smart TVs and Neopets and what else? What am I missing?
JOHN: NPM, Private…
PAUL: NPM, oh yeah.
JOHN: …Private IP.
PAUL: Man, yeah. So, we’ve had a very full signal and e-mail relationship but I had never had you on the show. Actually, this is for Episode 200, John.
JOHN: Wow. Well, congrats on that.
PAUL: Thank you very much. Talk about your work. Talk about all the work that you do ‘cause you wear many hats.
JOHN: Sure. I mean, my day-to-day I’m a blue teamer, so I’m an application security engineer which just entails configuring and managing application security tools. There’s a whole suite of them. One of the other things I do is I run a bug bounty program and I am actually writing a book about establishing and managing bug bounty programs, so that’s gonna come out in June of 2021 through Wiley.
PAUL: Wow. Are you like, thirty-three? Is this your Jesus year or something?
JOHN: No, no. I’m actually twenty-six, believe it or not. You know, a lot of people have said that and I really think it’s because of the pandemic.
PAUL: Yeah, right.
PAUL: It’s everybody’s Jesus year.
JOHN: There’s so many people that I’ve been collaborating with and just working with that [00:05:00] may not normally be online as much as they are, so it makes a lot of people bored, you know? Everyone wants to do security research and fun stuff.
PAUL: So, you blue team on behalf of a major online application front. I don’t know if you can say their name or not but I know who they are. I don’t know if you can share who they are.
JOHN: Right, yeah. I mean, everyone knows pretty much at this point, yeah. It’s ShutterStock. A really good company. I do enjoy blue teaming for them and working for them.
PAUL: Let’s talk about some of the projects you’ve been involved in just in the last six months or so. I think the one that got me to — introduced to you was the TCL Smart TV vulnerabilities which you didn’t discover; you were kind of brought in by Sitcodes who I think made the initial discovery, but you needed some help. Talk about that one. How did that all play out and what did you learn?
JOHN: Sure. Actually, Casey Ellis, he is the founder of Bugcrowd and he has a — forums that he uses for helping researchers get disclosure called community.disclose.io. I was kind of interested in his project and I was like yeah, I can help out here and there. I’ve helped people get disclosure before so I figured why not just give back and try to help out? So, I went on the forums and literally I think day one or day two of being on these forums, I see Sitcodes posting about needing some help, so I communicate with them. I’m basically like hey man, what’s going on? Show me the vulnerability, right? So, he shows me the vulnerability and I’m like okay, this is a problem. We should report this. He’s like dig in, man, dig in. I’m like, alright.
So, I start looking more and now this is where it gets interesting because at this point it probably could have been disclosed and we would have just — that would have been it. It would have just probably been one CVE, whatever, but I have blue team experience, right? I have a little bit of web app experience and some API hacking experience. I started digging through these directories and I found something called the Terminal Manager. When I looked at the code for the Terminal Manager APK, I started seeing these API calls where they were writing data, taking screenshots, recording and all this stuff, and sending it to a Chinese server. I’m like, are they sending it continuously? Is it happening on-demand? I don’t know but this looks like a backdoor. It smells like a backdoor. Then yup, and then the whole escapade started at this point.
PAUL: Talk about that.
JOHN: Yeah, yeah, sure. So, we didn’t want to say that it was a backdoor immediately, right? Because it definitely looked like one but every time something like this happens…
PAUL: It’s kind of a loaded term.
PAUL: Not a term you throw around lightly. But anyway, go ahead, yeah.
JOHN: Normally, if it was a application based out of the US, I probably would have been more skeptical and dug a lot deeper before I said hm, maybe this is a backdoor. But the TCL TVs are Chinese, so China has a history of backdooring products, especially IoT products. Then when I started to look up TCL backdoors, I found old stories of backdoors and I’m like, no way. This is probably just a backdoor reopened. Then that kind of started it but still, we were gonna err on the side of caution a little bit. But then things just started to get more and more suspicious so I called their support. Their support said they didn’t have a security team, asked me to just give them the vuln over the phone so they could put it in their Help Desk. I said no, no way. As you know, we started reaching out to people and then finally we did get in contact with their team, right? But they were just about as confused as us. It was just…
PAUL: Which is really like…if we could kinda pause and kinda parentheses this; TCL is like, a huge, huge company. It’s the third-largest maker of — seller of television sets in North America. It’s a huge Chinese conglomerate, electronics maker; phones, TVs, every — smart wearables. I mean, everything. This is a multi, multi, multi billion dollar global company. The fact that in 2020 or 2021 — well, it was 2020 — you call them up, contact them about a vulnerability and they say yeah, we don’t have a product security team. It’s just sort of like…
JOHN: It blew my mind, honestly.
PAUL: [00:10:00] It’s like your head explodes.
JOHN: It’s funny; a couple days ago I was like, what if I look at TCL news in China to see what they’re saying about it, right? So, I hopped on a Chinese search engine and I started translating some stuff. It’s kinda funny; they were like — it looks like — I don’t know if it’s the gov or who but they were pushing out some propaganda saying it’s not a backdoor, there’s no backdoor in the US. Technically, they’re right. Technically they’re right. However, they didn’t mention the backdoor in their own country.
PAUL: And in a lot of other countries, too.
JOHN: Yeah, Europe, Europe and just all over the place. They didn’t mention anything about that, though.
PAUL: Yeah. They kept saying that to me when I got comments from them too, as if that mattered. It’s like, Security Ledger is a publication based in the United States but it doesn’t particularly matter to me that the Android TCL TVs sold in this country didn’t have the backdoor application installed on them. I mean, you’re still selling hundreds of millions of them in India and other countries as well as China.
JOHN: Yeah, but then — I mean…
PAUL: It’s still a problem.
JOHN: No, exactly. But then take Sitcodes’ second CVE finding where he found that you could overwrite the privileges of the — or, sorry, overwrite the firmware because the privileges are too lax. Then what do you have? You have a TV in the US that you could overwrite the firmware and install overseas firmware on it. So, what do you have now? Now you have, okay, well, maybe there’s not a backdoor but do we know that for certain? I most certainly didn’t test every single US TV to see what version of the firmware that they had, right?
JOHN: I just know that they had backdoors.
PAUL: Yeah, and if you’re sort of one software update away from a backdoor, is that in and of itself a problem? You know, I mean I think the bigger question is potentially any smart TV these days has a camera and a microphone ‘cause they all have been — they all do voice recognition. They’re all kind of surveillance devices sitting in your living room. I guess the question that consumers should be asking is how do any of us really get comfortable that these things aren’t spying on us or couldn’t be used to surreptitiously surveil us either for commercial purposes or by authoritarian governments? Or non…
JOHN: It’s a loaded question.
PAUL: It’s a loaded — it is a loaded — those are my favorite kind. I don’t actually know the answer to the question but clearly what TCL is doing is not putting — giving anybody very much comfort.
JOHN: Yeah. I mean that for sure is not the answer, what they’re doing. But they did stand up a security program.
PAUL: They did, yeah.
JOHN: I should mention that.
PAUL: Yeah, and their PR people were very responsive to me once they kind of clued in and I think did take steps as far as I can tell. I mean, based on their statements. When you sort of look at the combination of a authoritarian government that clearly is very interested in technology-enabled surveillance on a mass scale not only in their country but elsewhere, making — with connections to this electronics multinational that’s churning out smart TV screens by the million, that probably should have people concerned. I don’t know.
JOHN: Right. Yeah. I think part of the problem is we just don’t have a good stance. Our entire government should have some sort of standard, right? I think we’re progressively getting there, especially with hacks like SolarWinds but it’s gonna take some time and I think that it’s gonna be needed at one point in time, especially if we’re gonna be importing electronics from other countries.
PAUL: But the other story I thought that you were involved with was the NPM private IP application. Just talk a little bit about that. That was actually in some ways much scarier than the TTL, than the TCL…
JOHN: Which is funny. Which is funny. It’s actually hilarious that you say that, right? Because — and I think part of why that got less coverage is because TV is scary, right? Every consumer has TV in-house, right?
PAUL: Absolutely. Yeah, yeah, yeah.
JOHN: That’s just like, that type of thing whereas NPM private IP, you tell that to someone that’s not in tech and they’re like, what? What does that mean? What does that mean to me? But it does mean something. Basically, what ended up happening was we had someone through our bug bounty program report an SSRF vulnerability. It was business as usual. We resolved it.
PAUL: [00:15:00] That’s Riverside Request Forgery for people who are not…
JOHN: Right. So, basically what it allowed him to do was interact with resources on the server, on the client side, AKA the side that he could see and interact with. However, he wasn’t really able to do much. We fixed it and then a few weeks later he bypassed it which isn’t too uncommon to get someone to bypass it because they’re clever and they’ll find bypasses occasionally. Then he finds a bypass again and I’m like okay, no, this is — no one should be bypassing it more than once. I take a look at it and then finally one of my co-workers, Harold, he actually — I’m like, we need to track this down and figure this out. He’s like hey, well, I figured out the logic for what they’re using to block and it’s a NPM private IP. So, he shows me that they’re using Rejex’s rules to basically filter out which IP should be blocked or which IP ranges. I’m looking at it and I’m going oh my god.
I’m like dude, this is a zero-day. He’s like, what do you mean? I’m like well, the entire package is built for preventing SSRF, right? Preventing threat actors from interacting with internal IP addresses. I’m like, but they’re not filtering out any sort of encoding on these IP addresses. They’re not filtering out different types of payloads. They’re only focused on just the Class A, B, and C ranges of IP addresses. So, I reached out to Mitre, got assigned an ID, reached out to the software engineer and he’s like yeah, I could see where this is a problem. He was like, can you help me fix it? In my head I’m just like, what? You want me to fix your package? I’m like alright, well, could I fix it? Probably. Could I fix it well? That’s another story. Then I brought in Nick Sahler.
PAUL: You reached out to the guy who created this tool, identified the flaw, and he was basically like hm, I’m not sure I could fix that.
Could you give me a hand?
JOHN: Yeah. Yeah, exactly.
PAUL: Which, I would consider a red flag.
JOHN: It was bad. I was like, this guy is not prepared for what he’s about to face, which was fine.
PAUL: We won’t name him but I mean, you can figure out who it is. But he’s a Montenegro-based developer. He had a number — has a number of different projects but certainly nothing — like, he was a developer. He was a solo developer and this was more or less his project. He worked on it basically himself, right?
JOHN: Yeah, yeah. It was just him on the project, so at this…
PAUL: Yet it was hugely, hugely popular. Talk about how widely used it was.
JOHN: I mean, it was extremely utilized as a dependency, surprisingly in cryptocurrencies and all sorts of stuff; hotels and banks, I mean, you name it. It was all over the place. I kinda laughed to myself a little bit when we were in the process of fixing it ‘cause like I said, I brought Nick Sahler in and I brought Sitcodes in to help this guy write a patch which they did a great job on. But when I was thinking about the impact of this, I laughed a little because I was like, I probably just made it a lot harder for a lot of security researchers because there’s probably so many SSF — SSRF vulnerabilities that have been discovered through bug bounty programs and paid out for and then bypassed and paid out for again, and probably even multiple times before people have caught it and said oh, maybe I should get rid of this package and use something else.
PAUL: You could literally…
JOHN: Which is funny. It’s funny.
PAUL: I just kind of — just, you know, if you were hip to this, you really could have — you could have made a pretty good penny just going around and reiterating and not addressing it at all.
JOHN: Honestly, honestly, we debated it but one of the — and just going through — I’m not gonna lie to you; we debated it. It was the big one.
PAUL: You could have made a ton of money doing this.
JOHN: But what I really thought about was just how — it would take time because nonetheless, you would still have to find a point of entry for an SSRF, right? You would [00:20:00] have to find a vulnerable part of the web application that is vulnerable to this SRF — SSRF which within itself is bug-hunting so ultimately, it wasn’t an easy win. It would have taken quite a bit of time to fuzz for all the SSRFs on all of these domains and look for it. Ultimately, we were just like you know what? Let’s just do the right thing here; cement this, get a fix. It was interesting. When you fixed it too, a lot of companies started dropping it as a dependency. I think more or less it was the idea of just fear, a little bit, right? Like, when someone gets hacked, they stop using that product, right?
PAUL: The bigger issue that that raises is of course just the open-source software supply chain issue. This is sort of where we were with Heartbleed in Open SSL a few years back and there have been a number of other incidents since then. But we’re living in a DevOps world where — and an agile world where developers, applications are really pretty much assembled from these various open-source modules and libraries and components. While we like to think that there’s a lot of security built in, this would suggest that a lot of people are kind of making the mental — doing the shorthand of saying well, it’s got this many dependencies, this many other people are using it. Somebody must have audited it so it must be trustworthy and I’ll use it, too.
JOHN: I kind of have a bleak outlook on this, right? I hate to be a Debbie Downer in any sense but application security is two aspects. All of your applications; web apps and mobile, so on and so forth and the second aspect, an overlooked aspect, is third-party integrations into the application. I think that gets overlooked a lot but it’s an important aspect, right? You’re talking about an application that may have full admin permissions to your environment. These supply chain hacks, especially when you’re talking about software engineering and different packages that can be built into tools that many enterprises are using but are open-source and they don’t know what dependencies that package is using, that’s a big issue and it’s a — that’s a huge part of the problem.
Then obviously, that second aspect is we’ve — we’re taking applications kind of into a new world, right? It used to be simple. Now, I wasn’t in tech when it was but now we’re at the part where application stacks are just getting exceedingly complex like integrations, all of the third-party aspects of the application that are in place. It’s almost like wow, just thinking about it, like operating systems alone; one application could be the make it or break it that ends up in a very widespread exploit. A good example of that is JetBrains. JetBrains, there was evidence or proof that led to JetBrains being compromised. You have to think; you know how many software engineers use JetBrains to develop their enterprise code? Right, and they’re just collecting all of that?
PAUL: You’re not just kind of kicking back and ruminating on this. You’ve actually stood up a new research group and it’s Sakura Samurai. Is that right?
JOHN: Yup. It’s a Sakura Samurai.
PAUL: Sakura Samurai. Talk about that. Talk about the name and talk about who’s in it and what your mission statement is.
JOHN: Sure, yeah. Sakura Samurai was kind of developed as a hacking group stemming with influence from Cult of the Dead Cow. What I basically had the idea for was just a hacking group that wasn’t a collective, for one. We don’t want to be a collective that’s just open to all because I think that the problem with that is you end up getting another black hat group. Not all hacktivism is bad in the sense that hacktivism has done a lot for our world, exposed a lot of manipulation. However, that’s just not the path that I saw for this group so I kind of made it very, very closed.
PAUL: You’re only ever as good as your most narcissistic, sociopathic member.
PAUL: A bad place to be.
JOHN: [00:25:00] Exactly, exactly. If Kirt’s listening, that means you. When it really comes down to it, I thought about — and when I say security research, what I really want to say is I really want to call it just what it is; hacking. We’re hacking. We’re hacking but we’re doing it ethically. We’re going after bug bounty programs. We’re going after companies that have vulnerability disclosure programs and we’re doing the right thing. We’re disclosing it to them. I know I hear a lot of grumbles, especially from members of black hat groups. They’re not happy about this group. They think we’re ruining hacker culture and I would disagree. I would say that the reason we chose the name Sakura Samurai is because ‘sakura’ in Japanese means ‘cherry blossom’ which in Japanese culture is kind of rebirth, you know? The idea here is it’s a play on words a little bit. It’s the rebirth of hacking culture. Yeah, I recruited some members that I trust. So, on that list, the lineup, we have Nick Sahler who actually founded it with me, so we’re both co-founders of this group together. He’s a software engineer. He goes by Arctic.
Then we’ve got Robert Willis. Robert Willis goes by Rejex, so he does offensive security and red teaming for the military. Then we have Jackson. Jackson’s a unique member. He goes by Kanshi. He is basically a fifteen-year-old hacker and a very talented one. A very, very talented one. I love that kid to death, seriously. He is a talented hacker. Yeah, so he’s great. He actually kind of started the whole United Nations hack debacle that we just pulled off. We have Ali Diamond, so she goes by ShÄde. She’s a software engineer. She’s currently working on some other stuff right now but she’s gonna be hopping in and taking a look at some of our projects real soon. Then we have the founder of Anonymous, right, Aubrey Cottle, AKA Kirtaner. It was — I didn’t expect him to join and it is very controversial. It’s making a lot of people angry.
PAUL: Talk about that. Obviously Kirtaner, Aubrey, a very storied past. Not all of it on the right side of the double yellow line.
JOHN: I was actually working on some research on Parler and researching their security vulnerabilities, kind of just poking for holes a little bit and just seeing what the platform was all about ‘cause everyone was complaining about it. Then I really started to uncover a lot about Q Anonymous and a lot of these conspiracies they were up to. It looked like the classic playbook brainwashing of society.
PAUL: Oh yeah, yeah.
JOHN: Then when I started to do more research, I saw FBI warnings on them, calling them a terrorist organization. I started to see all this kind of stuff and I was like oh my god, this is bad. I’m gonna research this more. No, I started looking into them a little bit, started researching. Won’t say too much but I found some stuff and I ended up messaging Aubrey and just saying hey, what if I found some stuff?
PAUL: Yeah, because he’s been very active and kind of trying to…
JOHN: Stop them, basically. Shut them down.
PAUL: …stop them, pull back the curtains, and yeah.
JOHN: Yeah, so I saw that he was fighting QAnon and I was like, he’s literally fighting a terrorist organization. So many people probably have misconceptions about this but I’m starting to see what he’s saying because I’m not on the surface level at this point. I’m finding these Q Anonymous domains. I’m looking at them, I’m looking at their methodology and their brainwashing. They’re uploading playbooks of just insanity, right, to try to target people and go after people. I’m like, this is bad. I’m like well, I should probably just pass this information on. Then I ended up passing it to him and a friendship kind of started there. It’s like, oh.
PAUL: You’ve got a couple pots on the stovetop right now as well, the things that Sakura Samurai is actively working on. Can you give us a little preview of what you guys have coming up?
JOHN: Sure. So, we have three things in the chamber right now. I want to mention — and this is a great example — these three major vulnerabilities that we [00:30:00] found; if anyone is questioning the ethics of our group, realize that we haven’t even leaked any of this. We found a ton of stuff and some of it is juicy. The media will want to hear about it, so one of them…
PAUL: I know I do.
JOHN: Yeah, yeah. I mean, so one of the things that I had found; I was actually doing some research on an application offered in a bug bounty program. It was a executable application. I wasn’t really getting anything or getting too far with it, and then I found a folder. I was trying to find the root folder that it was in and I saw an application in there that I had installed. It’s a privacy-based end-to-end encryption communication platform. I was just like…
PAUL: Associated with a very well-known service.
JOHN: Associated with a well-known service and I was taking a look at this and going huh, I wonder what’s in this folder. So, I just opened it up and I see a bunch of files and I’m like well, if it’s end-to-end encrypted, all these files should be encrypted or yeah, should just be a bunch of garbage data that you could do nothing with, right? Then I go in a folder and I find a privacy issue. Let’s just say that. At this point I was — I didn’t believe it to the point where I kept doing it over and over again and reproducing the flaw over and over again because I just was like, in disbelief. At this point I was like alright, who’s up? On our group chat for Sakura Samurai I was like alright, who’s up and who wants in on this? I had Jackson and Kirtaner hop in and they helped me look more and more and we found even more exposure with this application.
At this point I knew it was bad. The app has a — I will say either a vulnerability disclosure program or a bug bounty program. I’ll make it a little more difficult. So, I reported it and they were like ah, well, it’s kind of a minor issue and this issue isn’t quite what you think it is. Then I showed them more proof and they were like okay, yeah, this is an issue. We have to investigate this more. This is what we project the CVE rating to be. I got a CVE assigned for it and yeah, now I’m just kind of waiting on them to patch it so I could just drop it ‘cause this is something that the federal government can use against people. This is something that threat actors can use against people. I mean, there’s a lot of aspects here that are in play.
PAUL: Would this be a — would this allow man-in-the-middle-type attacks, would it allow just data theft or disclosure?
JOHN: Let’s just say if you knew this was — this type of data was being disclosed, you wouldn’t use the application. Black hats for sure wouldn’t use it and regular, everyday people wouldn’t use it if they found out about it because they would think that they were secure in operating the app as they were operating it. Same thing with security researchers, too. If I found out about this, I probably wouldn’t use the app ‘cause I’d be like well, how could I trust them, right? They’re affiliated with a major company.
PAUL: …local execution or not? You’d need local execution.
JOHN: Privacy-based local access. However, to a lot of people, especially bug bounty programs, local access flaws aren’t a big deal but I think it’s a giant deal for this company because their whole platform is privacy, right? It’s a privacy vuln. Yeah, it’s huge. I wouldn’t be surprised if either nation state threat actors were using this or if our own federal government was using it. I honestly don’t know. It’s all speculation ‘cause I have no way of proving it. However, it is quite concerning. So, that’s one thing that we found.
PAUL: Final question; you’re also doing a lot of work looking at government networks and resources. Just talk about that.
JOHN: Sure, yeah. The other two things we have in the chamber are two different governments in the same — are government entities in the same country. One of them is the entire federal government and one of them is a more localized area. [00:35:00] This is international government, by the way. We just found that both of these entities are just riddled with vulnerabilities. For the one vulnerability, actually Jackson, believe it or not; Jackson found over 500,000 or roughly 500,000 citizen records, let’s say. It’s really, really big stuff, really juicy. Massive, massive data breach. He disclosed it through their VDP. Honestly, I don’t know how long it’s gonna take them to fix it and that’s not considering some of the other bugs that he found too, like credential pairs and…
PAUL: What is the underlying problems that these governments are having? This would probably include the US government and the state governments as well but these are international governments?
JOHN: Yeah, international governments.
PAUL: Those are common regardless of whether you’re talking North America or other countries.
JOHN: Yeah, very true, very true. It doesn’t matter what type of government you’re talking about. The problems exist all over and that’s I guess a good segue into the second government issue we found which was — oh man, I mean, it’s a lot of stuff. We’re still — it’s gonna take us a few days to really go through all this kind of stuff but at this point we’ve acquired medical records, undetermined amount. We’re still assessing the impact. Probably seventy or more exposed Git files where we were able to dump the projects on the domains, probably seventy — roughly over fifty credential pairs for databases. I mean, the problems go on and on; police reports and records that aren’t exposed publicly. It’s like, yeah, it’s a huge deal but we’re trying to assess the impact and get it patched before releasing it because we don’t want more people attaining the data than probably already have.
PAUL: John Jackson of Sakura Samurai, thank you so much for coming on and speaking to us on Security Ledger Podcast.
JOHN: Of course. Thanks for having me. I really appreciate it.
PAUL: John Jackson is the co-founder of Sakura Samurai and an independent security researcher. Up next, one of the lesser-reported subplots in a recent SolarWinds hack is the use of stolen or compromised digital certificates to facilitate compromises of victim networks and accounts. Stolen certificates played a part in the recent hack of Mimecast, for example, as well as an attack on employees of a prominent think tank, according to reporting by Reuters and others. How is it that compromised digital certificates are falling into the hands of nation state actors? One explanation may be that companies are managing more digital certificates than ever but using old systems and processes to do so. The result; it’s becoming easier and easier for compromised or expired certificates to slip through the cracks. Our final guest this week thinks we’ve only seen the beginning of this problem. Avesta Hojjati is the head of research and development at the firm DigiCert.
As more and more connected things begin to populate our homes and workplaces, certificate management is gonna become a critical task he says, and one that consumers and even businesses are ill-prepared to handle right now. What’s the solution? Hojjati thinks that more and better use of automation may be the answer. In this conversation, Avesta and I talk about digital transformation and how the growth of the Internet of Things is raising the stakes for proper certificate management and why companies need to be thinking hard about how to scale their current certificate management processes to meet the challenges of the next decade. To start out, I asked Avesta how the covid pandemic and the shift to remote work has impacted how PKI infrastructure and digital certificates have been used and deployed in the last year.
AVESTA: Sure. My name is Avesta Hojjati. I’m the head of R&D here at DigiCert.
PAUL: Avesta, welcome to Security Ledger Podcast.
AVESTA: Thank you for having me.
PAUL: I mean, obviously the big story of 2021 is the covid pandemic and in the context that we’re having this conversation, just the huge changes that the pandemic has forced on the workplace, right, and on how companies operate. Some of those were changes that were already happening; maybe slowly, but they happened a lot quicker once covid came around. How has PKI played [00:40:00] into that and I guess I mean how have you at DigiCert noticed consumption of your service and utilization of your services changed at the onset of the pandemic, and has covid kind of identified or exposed some new PKI use cases that maybe we hadn’t thought about before?
AVESTA: Certainly, yeah. As you mentioned, 2020 started by having only one story and that being covid. Immediately, the majority of the workers started to start working from home and the remote work by itself went on a very incremental increase. Looking at that, we saw a pretty high increase of certificate utilization when everybody had workers where they had to work from home. They had to receive a certificate on their personal devices and that was the method for them to authenticate back to the workplace. Utilizing VPNs or virtual private networks obviously increased tremendously and that by itself does utilize some portion of PKI. The other portion of this for regular consumers was that we noticed a good amount of new e-commerce websites are coming up. Restaurants that before didn’t have a website, they started having websites and they were able to have their menus online and users could have ordered the food from their website at this point.
In all of the scenarios, you needed to have a secure method because like any other global problem that we might see, you usually have adversaries who are going to take advantage of these problems. Again, covid being the prime example of that. You will have adversaries who will start sending phishing e-mails and enterprises and smaller companies, SMBs especially, they had a way to actually be able to increase their security. Certificates was an easy, affordable, and a scalable solution for them. In 2020, we actually saw a pretty good increase when it came to certificates. The exact figures aren’t out, obviously. We just started 2021 and we are still analyzing those data but compared to previous years and employees going remote, more e-commerce websites are coming up. We saw more secure connections being established.
PAUL: As we’re speaking right now actually, the Consumer Electronics Show is going on in Las Vegas kind of, except it’s mostly remote. Also, it’s not the usual spectacle that it is, but brings to mind that there’s also a huge explosion in electronics, smart and connected devices, and obviously with some of the other things we’ve talked about; covid and the shift to work from home. A lot more of those devices are finding their way into workplaces. Talk just a little bit about how this Internet of Things explosion is impacting the work that DigiCert does.
AVESTA: Let me start with the big trend that we saw in 2020. As more telehealth started rising up because again, people rather stay home instead of having that physical interaction due to social distancing, what we saw was a good amount of medical devices were being shipped to consumers instead of being utilized in hospitals. Obviously, considering that most of these devices are connected devices, the security of those devices needed to be taken care of properly. One trend that we saw was shipping these devices where they require to have certificates, where their database or firmwares need to be updated and they’re required to be cryptographically signed.
That was one area that we saw a pretty big focus on. Regardless of that, we actually have seen another area which is general IoT devices. As any other individual who, they at home usually try to do certain things around their house; improve it, change certain things. In my specific case, for example, my IoT device consumption in 2020 compared to 2019 was much higher. I acquired new devices to be able to automate certain things around my house and obviously that by itself just shows one sample but I’m sure the rest of the society are feeling the same where they’re able to modify their property, be able to add new IoT devices.
PAUL: We’re talking about this huge increase in the sophistication of the IT environments that organizations are managing, the growth in just the number of certificates out there that organizations need to manage, and also a higher bar in terms of how they manage them. You can’t kinda set it and forget it. Maybe you could twenty years ago. That all speaks to the need for more automation and this is a big meta trend in security as well; few hands and a lot of work. So, a lot of people are looking for ways to automate, particularly road security work. This applies in the PKI space as well. Could you talk to us about when we talk about automating some of these core functions, where there’s room for advancement there and where some of the investment is right now in automation?
AVESTA: Automation is a very big topic for us here at DigiCert. For the past couple of years, we have been focusing on this topic specifically because we realize that the problems that automation is able to solve. One problem being specific topics related to compliance. Our specific [00:45:00] industry evolves almost on a daily basis around compliance and once there is a problem, obviously you need to go ahead and replace that certificate. You can consider an enterprise with tens of thousands of certificates all on different platforms, in different regions, and often it’s a cumbersome approach for them to be able to replace every single one of those certificates individually. The approach that we have taken is to be able to automate this process via automated certificate management where you are able to log into our CERT central console, you are able to have a bird’s eye view over your organization as far as what certificates are being automated, which one of them you are able to automate, which one of them you are able to have a bulk issuance or installation.
As we have seen again in 2020, covid proved that automation is as important as anything else in your infrastructure. Example is if you have an IT admin who for any reason might be diagnosed with covid and he can’t work for the next two or three weeks, there are a number of certificates that are going to expire during that period. Now, basically it’s very hard to do anything because that admin is not there. He has to delegate some other individuals. In other words, we require to have proper planning. Whereas if you have automation deployed, really that human factor by itself becomes a more role of a policy maker and a manager instead of individual who has to go ahead and replace those certificates.
PAUL: Yeah, I mean one of the challenges I would think, and particularly as you’re moving to these kind of networks of things is that not — first of all, you have so many different types of devices out there that might have certificates that are part of their constitution that need to be managed but where you don’t necessarily have very good visibility into those devices. They’re not traditional IT assets. How does DigiCert see that problem working itself out to where it’s not just your web server and your web applications that need to be — have their certificates updated but it’s the refrigerator in the employee commissary and, you know, door-badging system and coffee maker and those types of things?
AVESTA: Definitely. This is where we are taking the — I would say a very old philosophy whenever we are discussing security. Many, many years ago, DARPA made a recommendation known as security by design. The idea was that once you are designing an application or a device being hardware, you need to think about security. Taking that approach, what we have been working on is a number of different methods which allows us — especially when it comes to IoT devices — to be able to be embedded within the process while you’re designing your IoT device. If there is a need for a certificate on that specific chip set, we do have capabilities to be able to give you that certificate installed on that chip set and once you are done installing the application and you’re sending it to production, you already do have the basis of security. Again, it’s quite important to look at this from a number of different aspects because hardware is only one part of this story and obviously IoT devices are everywhere.
But if you look at it in the modern DevOps when you have your continuous integration and continuous delivery, the same scenario applies. Instead of writing the application, building the application, and then requiring a certificate for that or any other security requirements that you may have, the developer by itself could think about adding that certificate into the build process. It could go ahead and utilize one of our APIs where we have integration to Chef, Puppet, Salt, and many other DevOps orchestrations, and you are able to receive a certificate and deploy it in production. Again, I think this is more than just a certificate. If you have to go by the philosophy that as I mentioned DARPA had, security by design covers every aspects whenever it comes to security and application development.
PAUL: I think you’re right on. I think that’s kind of where it needs to go. I guess how to you evangelize that down into the development community and — I’m thinking kind of CES and if you go and read the brochures and so on of the products on the floor of CES, there’s just not a lot of focus on security. There aren’t a lot of details for many of these connected products about the underlying security of communications and data that they might manage, even for very sensitive products, medical products and things like that. How do you nurture that security, that — what we saw at Microsoft twenty years ago, the sort of secure computing memo, that type of focus, singular focus on security within some of these development organizations where time to market and usability and features might be the focus?
AVESTA: I think the unfortunate truth about this is often security will be sacrificed for either usability or for go to market timeline. As you mentioned, the majority of the manufacturers are trying to rush for the process of having a product that can be the first in the market and for that, security by itself will be sacrificed. Obviously, later they will pay a very high cost for that. In order for [00:50:00] us to solve this problem, we have taken two approaches. One is every single customer for us is a unique customer where we have different type of products available for them and we try to be as part of their product development process instead of only being a provider. You can think about every single customer of ours being a partner. Once they decide to have an application or a hardware where it requires a certificate, we will be part of their design process. We provide our guidance as well as run a number of different POCs and that’s part of a job that some of my team members are doing as part of the R&D team which is looking at the specific use cases and they try to help customers to have the right approach for their application.
The other portion of it is regulations. I think this is somewhere that as an individual entity, we are able to work on and we are able to help but the enforcement of this comes down to regulators. It comes down to governments. Especially, you mentioned medical devices; back when I was getting my Masters, I spent about two years working on infusion pumps and I read almost — documentation about infusion pumps to identify what type of a security guideline FDA provides for these specific devices, FDA and FCC in combination. The language by itself, at least many years ago, was very vague. You need to have the security. This is the security level that you need to have. If that language is vague, it makes it much harder for the manufacturer to be able to actually understand it let alone being able to enforce and develop that.
PAUL: As you know, of course, there have been demonstrations of cyber attacks on infusion pumps, right, at Black Hat and other shows as well as other types of interventional medical devices. That’s interesting. We talk a lot in security about the shift left, right, about how security’s moving closer left into the development pipeline as opposed to right into production, post-production. Is DigiCert shifting left as well, and what does that look like?
AVESTA: Absolutely. We are trying to get to — I probably would call it zone zero and anything else. As I mentioned, security by design has been a philosophy for us and we are trying to promote that. In at least the products that I specifically cover, our goal has always been to be able to integrate a number of different third-party providers, especially as far as DevOps. I mentioned a couple of names; Salt, Chef, Puppet, Copperhead which is more…
PAUL: Those kind of key development tools and platforms, right?
AVESTA: Exactly, exactly. The closer that we are able to get to those providers, the more functionalities we are able to provide and this has been, again, a philosophy for my team and for the products that we are working on, especially when it comes to automation. We are trying to make it easier for developers to be able to acquire security services. Obviously, our goal now and in the future will be to focus on being closer to that ring zero and being able to have the ground of the security and start building on top of that.
PAUL: One of the big challenges always is there’s just so much legacy investment, so much legacy technology out there in enterprises, especially even in high-value sectors like banking and finance. Oh my gosh, you know, thirty or forty-year-old technology sometimes. How do we push those people onto more modern platforms, get them to let go of some of that legacy investment and kind of level up their security around certificate issuance and management?
AVESTA: Absolutely. Transformation doesn’t happen overnight but what could potentially happen overnight is a big catastrophe where your infrastructure could be down because you did not spend the days upgrading and changing and having a very solid infrastructure. We do see customers where they have thousands of different workflows and often they don’t want to get rid of some of those workflows because they are so invested on those because they believe the customer downtime is going to hurt them on the short run compared to the long-term of upgrading those infrastructure. On the other hand, as I mentioned, moving towards more DevOps applications such as Kubernetes, dock rising the applications, going more on the microservices, architecture is the new trend. In the customers that we have seen where they have very old infrastructure, often what we try to do is we try to stay with them as much as possible.
Again, our number one priority has always been customers, has always been to have very solid uptime and be able to provide their security needs. What we have seen again as far as trend is customers are start approaching their infrastructure and their distributor and all this in a sense of what is known as bucketing. You try to bucket your application based on the language that you have, how old that application is, and what the business case might be. Obviously, security is slowly getting into those areas. Again, this is not going to happen overnight to get it off the old infrastructures. We are seeing different trends in the industry where you are able to transform the application either by [00:55:00] rewriting them or changing your infrastructure approach whenever it comes to designing your next generation platforms.
PAUL: Okay, final question; I’d be remiss if I didn’t ask you about the big security story of — at least of 2021 which of course, the SolarWinds compromise. That had to do with a compromise of their build process and then some malicious — some backdoored updates getting pushed out to more than 10,000 customers. Obviously if a sophisticated adversary actually infiltrates your build process, that’s difficult not to crack, but when you look at the bigger picture questions around supply chain — software supply chain, is there a role for PKI? Is there a way — are companies either not managing PKI as effectively as they could or not applying it as effectively as they could to secure software supply chain or is there a way that PKI could be used to shore up some of these software supply chains?
AVESTA: Definitely. Let me start with — it’s quite upsetting to see that you’re still dealing with supply chain security, that this problem is evolving over time. But you mentioned two very specific points; one of them was do enterprises have the capabilities and expertise to manage their PKI in order to have a more secure supply chain? Often, the answer is no. At DigiCert, we have been doing PKI for quite some time. Let’s say since 2003, we have been in this industry. We do PKI and we do PKI right. But often what we have seen is others will pick up either open-source projects, they will purchase PKI, and they will assume that all that they need to have. Nurturing a proper PKI solution, especially when it comes to supply chain, is very important. Being able to have proper code signing for your firmware optics where your keepers are being protected on paper could be very simple but in theory is very hard to manage. What we have done here at DigiCert is to make that process very easy to be able to utilize. Again, my hope is that at least in 2021 after hearing about all these problems and attacks on supply chains, the industry can take a different look at utilizing PKI to secure their supply chain and manage their product.
The other portion of your question and the point I didn’t mention was is PKI the right solution? Especially when it comes to supply chain security, the answer is yes. PKI has the scalability, there are a number of different deployment models as far as being on [inaudible] or on the Cloud, and obviously the solution has been around for quite some time and it has proven to be useful especially when it comes to use cases such as updating firmwares, sending packages where they require to be cryptographically signed. The last word that I will add is for the folks who are listening, we need to focus on automation in 2021. I think 2020 gave us enough reasons to think about that. We need to start delegating certain things to applications where they’re running over and over and they’re doing one thing and they’re doing it right. Especially when it comes to certificate, I think automation is essential. Automation is necessary and having proper scalable and functional automation solutions in placed, it’s definitely the beginning of your PKI journey.
PAUL: Avesta Hojjati, thank you so much for coming on and speaking to us on The Security Ledger Podcast.
AVESTA: Thank you for having me.
PAUL: Avesta Hojjati is the head of research and development at the firm DigiCert. You’ve been listening to The Security Ledger Podcast. This week’s podcast was sponsored by DigiCert. DigiCert is the world’s premier high-assurance digital certificate provider, simplifying SSL, TLS, and PKI and providing identity, authentication, and encryption solutions for the web and the Internet of Things. Check them out at digicert.com.
Transcribed by: www.leahtranscribes.com