Episode 253: DevSecOps Worst Practices With Tanya Janca of We Hack Purple

In this Security Ledger Podcast interview from earlier this year, Tanya Janca of the group We Hack Purple (now SemGrep), talks with Security Ledger host Paul Roberts about the biggest security mistakes that DevSecOps teams make, and application development’s “tragedy of the commons,” as more and more development teams lean on open source code.

[Video Podcast] | [MP3] | [Transcript]

---

Editor’s note: since recording this conversation with Tanya, We Hack Purple was acquired by Semgrep, where Tanya Janca in now the Head of Community and Education.

One of the thorny problems facing modern development organizations is the gap between their development- and application security teams. In many organizations, application develop happens separately from application security testing including pen testing, red teaming and the like. That can create bad dynamics, with appsec teams playing the role of gate keepers and finger wagging disciplinarians, rather than collaborators.

Hacking Purple to Bridge The Dev-AppSec Divide

Our guest this week, Tanya Janca, set out to bridge those divides. The founder of the group We Hack Purple (recently acquired by SemGrep), Tanya is a skilled developer and experienced pen tester/red team-er who has always taken it as her mission to not just identify security weaknesses in applications, but also to work constructively with development teams to address those weaknesses and to develop the secure coding skills and habits to stop making the same mistakes time and again. The organization she founded, We Hack Purple, offers courses for developers to learn core application security concepts and skills, and offers discussion groups where developers can seek help from the community around a range of issues. (Tanya also hosts her own podcast, which you can check out here.)

Attacks on APIs demand a Security Re-Think

DevSecOps Teams’ Worst Security Fails

In this conversation, which was recorded ahead of the RSA Conference back in April, I asked Tanya to dig into the details of a talk she was giving on “DevSecOps Worst Practices.” That was based on her experience advising development and DevOps teams – things like failing to tune your testing tools and breaking builds under a tsunami of “false positives.”

Supply Chain Hackers LofyGang Behind Hundreds of Malicious Packages

Tanya and I also talk about some of the bigger threats to application security. Among them: threats and attacks on open source software supply chains and a “tragedy of the commons” playing out in the open source software space, where development and appsec teams’ heavy reliance on open source software and tools hasn’t spurred greater investment in- and contribution to open source projects. Tanya talks about the need to cultivate more “open source angels” to right the ship.

Video Podcast

Transcript

​ [00:00:00]

Tanya Janca, We Hack Purple: Tanya Janca, CEO and founder of We Hack Purple.

Paul Roberts, The Security Ledger: Tanya, welcome to the security ledger podcast. Welcome back. I believe I think we’ve had you on as a guest before.

Tanya Janca, We Hack Purple: Yes, thank you so much for having me again.

Paul Roberts, The Security Ledger: It’s great to have you. So I reached out to you, Tanya, to talk about this notion of worst practices in DevSecOps. But before we did that, I wanted to introduce you to the audience. And just get the Tanya Janca origin story.

Tanya Janca, We Hack Purple: So I was a software developer so I started writing software in high school and got a job at a big tech company while I was still in high school. Then went to college for that.

But I was also a professional musician. So I would play in bars and music festivals all over town. I [00:01:00] toured a very tiny bit.

I released a whole bunch of albums. And then so as a software developer, like I found great work life balance compared to InfoSec and and I love writing code. So it was great. But after enough time one of the organizations I worked at hired a pen tester. And he was in a band. So obviously our bands had to play together.

That was the only thing that could happen. And we became friends. And for the next year and a half, he bugged me to become a pen tester. He’s you would be so good, man! Come on, man!

Paul Roberts, The Security Ledger: That doesn’t sound like a a Canadian “accent ,Tanya!

Tanya Janca, We Hack Purple: I !Know, I know. But eventually he convinced me to apprentice under him for a year and he gave me some of my first contracts.

And then and then I ended up getting a job doing pen testing full time and within about a year, I was like “Oh, I just, I want to stay. I don’t want to leave yet. I need to help you fix this.” And so my next professional [00:02:00] mentor, Sharif Kousa, he runs softwaresecure.ottawa. And so he, had been doing AppSec forever and he was the head of the OWASP chapter.

And he made me one of the leaders of the OWASP chapter in Ottawa. And he said, Tanya, for the past year, you have been doing AppSec, not pen testing. You do a pen test, but then you’re threat modeling with them. You’re like evaluating their architecture. You’re doing all this stuff that’s not your job.

That’s why your boss is always on your butt about how long you’re taking. And that’s why the clients always keep asking for you. And so you could just do this full time. And it’s called AppSec and it’s a job and it pays just as well. And he helped me move into that full time. And then I haven’t looked back since.

And you and I were talking just before we started recording. I’m very, extroverted and I really love people. And my favorite people are software developers. When I was in high school, my parents are like, what are you going to take? Cause I got an award for basically everything I got, I was on the honor roll and all that.

And I got accepted to every university and [00:03:00] then some and I was like, I like the programmer people in my class the best. So if I’m going to go to school for four more years, I want to go school with just them. And he’s you get to work with devs all day. You you get to have lots of social time.

You’re not stuck in a cold data center, or you’re freezing your buns off. And I was like, oh, this is way better. I’m doing this. And so I have not looked back. I’m just like, yay, AppSec!

Paul Roberts, The Security Ledger: What’s really interesting too is like that pen testing and development skill set those are often actually discrete things, right? Which is often why the pen tester kind of finds what they find and then they’re Not my problem, you know figure it out so it actually to have somebody who can both find the problems and then be able to talk about how to fix the problems is a sort of unique thing.

Tanya Janca, We Hack Purple: I guess that’s how I started We Hack Purple. We do secure code training appsec training. We help people build appsec programs and we have this [00:04:00] big online academy and then we do live training as well. And we have this huge online community that’s free.

And there’s courses in there for free. There’s events constantly. and just everything in the WeHackPurple community is free. And so we actually just launched four more professors. And so it’s pretty exciting. I have to say seeing your business grow and seeing just the community explode. Like we have 7, 500 people in there now that we all do stuff together. And yeah. And I’m not the star of our events anymore. Now the community members have been the star for a long time. And so it’s it sounds very basic saying for me to get to know all of them so

Paul Roberts, The Security Ledger: right Talk about the significance of the “purple.” So we hack purple is at wehackpurple.com explain the purple part of it.

Tanya Janca, We Hack Purple: Okay. I, so I, started and I was a coder and then I started doing pen testing and I thought I really want to be a pen tester because I thought there were only three jobs. There was [00:05:00] the firewall person. There is the risk compliance person and there’s the pen tester. And that’s all I thought existed.

So I was like one of them’s more technical than the others. I’ll probably be more comfortable with that. And then obviously I discovered AppSec but red teaming is offensive security. And so when you’re a pen tester, you’re a red teamer. And then when I started doing app sec and I kept helping people fix things, they’re like, Tonya, that’s blue team.

You’re acting on the wrong team. And I was like no, those guys are great too. And so people started saying, oh, you can’t make up your mind. Whether you want to do. Offensive, and I don’t mean swearing at people, aggressive testing of limits, offensive security, and then defensive security.

And so people start saying, oh, you’re a purple teamer because you keep doing both. And so when I went to make my Twitter handle, which was during the WannaCry outbreak, my email address was shehackscomputers and that was too long to have as my handle. And so I [00:06:00] was like, I don’t know, “shehackspurple”? Whatever. It’s not like anyone’s ever going to follow me because I just intended to follow other people.

And then eventually I started getting more followers and doing more things and people started referring to me as the purple lady. And people would joke like, Oh, you’re wearing a red dress. You’re not wearing a purple dress. What are you doing? So I started wearing purple shirts and purple dresses, and then I actually put purple in my hair for a long time and people just loved it. So when I. Start my company people are like, what are you gonna name it?

I’m like obviously it’s We Hack Purple: What else could it be?

Paul Roberts, The Security Ledger: Prince fans along the way as well. I don’t know it’s possible

Tanya Janca, We Hack Purple: I’m pro-Prince!

Paul Roberts, The Security Ledger: I am pro Prince, as well.

Tanya Janca, We Hack Purple: Yeah, let’s let’s do

Paul Roberts, The Security Ledger: No So you’re doing a talk at the RSA Conference in San Francisco and you you Your, subject, is DevSecOps Worst Practices. Just give us a little preview of some of the stuff you’re going to be talking about, Tanya.

Tanya Janca, We Hack Purple: Awesome. So since 2018, I have been doing work with [00:07:00] IANS research and meeting with one or more appsec teams per week since 2018. And so I always take notes and I try to track trends. And so I wrote this talk from basically doing five years of research with different appsec teams. And so I’m going to present 15 fails that I’ve seen over and over and over and like the first one is super obvious and it’s breaking builds on false positives, right? Like you don’t want to be the boy that cried wolf. You don’t want to be the person that’s like constantly like ringing the alarm bells when there’s nothing wrong.

And so I recommend to avoid this. Practice first. So I usually like the first time I get a tool, I’ll just copy some, like someone else’s working repo. I’ll remove the release components. So it’s not actually releasing anywhere. And then I’ll test the tool and I’ll tune it and I’ll make sure it looks really good.

And it’s going fast and it’s not breaking for no reason. Then I’ll show the team, get their permission. Then I’ll implement it the same way, [00:08:00] but just alerting mode. And I’ll make sure that it’s not at a spot where it would break stuff. falsely. So for a few months, I’ll just go through and be like, okay that was a false positive.

This wasn’t et cetera, et cetera. And then once we have the things fixed, so they’re going to pass. I just start breaking maybe on criticals first, then eventually highs than mediums. And so a lot of companies, they literally, they put it in, they flip it on in a default mode. No tuning, no testing, no training, and they’re like, break, break, it just breaks constantly.

And so just not testing the tool, not tuning the tool, all of those things.

I, I find that like those tick people off a lot. Another one that I’m going to present, it took me a while to spot this as a trend and realize this was something we could fix. And that’s. So I call it missing test results, but what I mean by that is, so the tool runs a bunch of scans and it breaks or it doesn’t break, but when it [00:09:00] doesn’t break, they’re just like, “Oh, cool! I’m perfect.” No, you’re not perfect. The cool static analysis or software composition analysis or dynamic, whatever it is. Whatever test you’ve run, your secret scanner, your linter. It hasn’t put the results into your CI/CD artifacts. It hasn’t sent you an email. It’s walked away in that system. And so quite often only the AppSec team even has access to that system.

And so it has thousands and thousands of bugs in there. They aren’t in your bug track. They aren’t getting emailed to the devs. The devs don’t see it. Like, when you finish your CI/CD, it tells you stuff. And it’s not in the list of things it tells you. It’s not by the way, there’s an artifact here.

And that means a report from the security tool. And just because you pass the CI/CD, doesn’t mean that there’s not work I would like you to do. Does that make sense? It’s not critical. I’ll still let you go to prod, but I really would like this to be in the backlog for [00:10:00] your next sprint.

And it’s just not in there. And the AppSec people are like, why are they not fixing any of these bugs? And the devs aren’t like, they come by it, honestly, they think everything’s fine, because no one’s told them otherwise,

Paul Roberts, The Security Ledger: right. Mm hmm.

Tanya Janca, We Hack Purple: to this other tool, or even worse, so another one.

Where you have six different testing security testing tools and then all these dev tools they’re expected to log into six dashboards.

Paul Roberts, The Security Ledger: Mm hmm. Mm hmm. Right,

Tanya Janca, We Hack Purple: wants to

Paul Roberts, The Security Ledger: right,

Tanya Janca, We Hack Purple: Oh yeah, I just have to check my 25 dashboards. I’ll be back in

Paul Roberts, The Security Ledger: right.

Tanya Janca, We Hack Purple: Okay, one more. So I call it “runaway tests”, but I’ve seen this over and over again. So you use up all the resources. So then no one else can run tests. So I’ve seen it where like a, I remember I was working somewhere and we bought 3 engines for this AppSec tool. I won’t name it. It’s a good tool but we had 25 teams [00:11:00] and it took, so they told us, oh, we’ll take, it’ll run in 15 or 20 minutes.

Meanwhile, the AppSec people have flipped it on and it was running sometimes 7 hours. And so a whole engine would be used up for the whole day. And so that means other, the other 19 teams had to share those two engines. That should be fine, right? No, it wasn’t. Another one took four hours. Another one took an hour and a half.

And so the devs would go and kick off their CI/CD and it would say “waiting on resources… And they couldn’t get to prod. And so then guess what happened? They just started flipping it off. And meanwhile the vendor or the sales person is telling us, oh, it should run in 20 minutes maximum, but usually 7 to 15 minutes.

I’m like in reality, what it’s doing is sometimes running a whole day. And so then the dev, the developers are stuck waiting with resources. They can’t do any of their other tests that they need to see. So they’re like, oh, I’ll just flip it off for this run, but I’ll flip it back on later. And they don’t remember. [00:12:00] They never intended to

Paul Roberts, The Security Ledger: It’s become a bottleneck. So they’re gonna get around it to do what they need to do. Yeah,

Tanya Janca, We Hack Purple: Yeah. And I was like, I can’t buy 20 engines. I cannot afford that. I’m going to have to fire my whole team and we still wouldn’t be able to buy 20 engines. Those three engines were $160,000 a year. And I was just like, I can’t imagine trying to buy 17 more. I just like, I can’t even fathom. So I was like, we need to find a way to play nicely together.

And so we found ways. To do it, but so these are some of the things where I’ve seen over and over again, where like this runaway test, it’s not only that it takes a long time. It’s the other people can’t do their work. Then,

Paul Roberts, The Security Ledger: And what was the discrepancy between what the vendor thought, how long the vendor thought the testing would take and how long it took in reality?

Tanya Janca, We Hack Purple: I don’t know what app they tested it on, but we were just testing the delta. Just like we changed [00:13:00] 20 lines of code or maybe 100 lines of code, which to me is not very much. And if they said it would take 7 to 15 minutes, but it generally took a minimum. Minimum 45, but usually around an hour and a half and I was like that’s a really long time because in, so while the devs waiting, they’re going to write other code and do other things and quite often they’d be ready to check in their next code before it was done running our tests.

And so then on top of that, once a week, or so, once every 7 runs, they suggested you do an entire code base scan. So not just the changes you’ve just made the skin and now it takes 7 hours, usually sometimes some of our apps, it would still be running the next morning when we’d come into work and I was just like, I’m sorry, guys we can’t do this.

And so we took, we removed the full scan and we started running those Friday nights and we didn’t put in a pipeline. So devs could still run those [00:14:00] tests and we changed it to only ever be the Delta and we ended up. This is going to sound really bad. We ended up just not renewing with that tool.

And we traded that static analysis tool for a startup static analysis tool. And it would, it was like seconds. So they would scan the whole code base in a few minutes, so it could cross compare things.

And developers would receive a report in their inbox 10 to 15 minutes later with just their bug that they just made, and I would receive a daily report of all the different newer bugs and all the old bugs and which ones were fixed. And I had this beautiful dashboard where we could see multiple tools. And basically. The previous appsec person was like, our CISO had insisted on signing a three year deal with these guys.

And I’ve been dealing with them for two and a half years. so then when we didn’t renew there was a lot of like drama with the vendor. They’re like, blah, blah, blah, this and that, but we [00:15:00] convinced the CISO to change and we did. And the devs were so happy. Yeah, having it run and just so it would pass them within three to seven minutes.

Which was so much faster, but receiving the report in 20, that was the thing that, that sealed the deal.

Paul Roberts, The Security Ledger: From your own experience coming up doing software development, what did you learn about application security or secure coding during your own education?

Tanya Janca, We Hack Purple: So I went to college in the 90s and there was no security whatsoever. There was nothing And when I was a developer until maybe 2012, I’d never heard any peeps About security at all. And so 2012 was the time where the security team where I worked, they got a VA scanner, a vulnerability assessment scanner, or what we call now, usually a dynamic scanner or a DAS dynamic AppSec testing.

Yeah. And so basically he had run it against my app and emailed me a report and [00:16:00] said, you can’t go to Prague because you have all these vulnerabilities. You’ve done a really bad job. And I was like, Whoa, what is this? And I remember looking in Google and finding and just having to spend a long time figuring out like, what is cross site scripting?

How do I fix this? There was, I think, two links. That Google turned up at that point. And I was just like, Oh man, calling my friends, figuring out what to do. And it took me three tries to pass the DAST and fix all the things. And I remember the, guy, and I remember his name, but I won’t say it. And he was, like, wow, that was so hard.

That took forever. I was not expecting that it made me late for my project. And he was like if you were a good dev, you would never have had those vulnerabilities in the first place!” And I like looked at him and I was like, “You may go hide in fear now!” I’m like, I was so ticked. I was like, excuse me but then I became a pentest -er just a few years later and I realized that guy didn’t know anything he, cause I was [00:17:00] like, can you help me with this? And he’s you should know. No, he said that. Cause he felt super insecure. Cause he had no idea how this thing worked. He had no idea what the report meant yet. He didn’t know how to read code or write code for that matter or fix a bug.

Paul Roberts, The Security Ledger: Right.

Tanya Janca, We Hack Purple: He chose me of all of our devs. To test me first, right? And I guess they’ve been hoping of all of them, maybe she’ll pass. And I didn’t and so then they knew like I would teach because I mentored all the junior devs and intermediate. So then I had to teach everyone about it. Then I started like a security training program for us. But I do think that he thought I might not bite back. I was like, “You’re being rude! That’s not okay”. And he’s you should have known this.” I’m like,” where, would I have known that? You certainly haven’t helped me and neither has your team.” I’m like, “what if you came over here and you were constructive for a change?” And then he ran back to the little security area.

Paul Roberts, The Security Ledger: And now an instructor, you know the importance of not [00:18:00] shaming people but, bringing them along.

Tanya Janca, We Hack Purple: I also learned that the developers were taught to do it this way. Like the very first lesson that every developer learns in every language is hello world. And so the first thing you learn how to do is put hello world on the screen. And the next thing you do is you say, what is your name? And so you learn how to take their name and say back.

Hello, Paul, right? And so you learn to do that. And what they do is they teach you how to do cross site scripting. It’s the perfect recipe. We take input from the user. We don’t validate that it’s what we’re expecting whatsoever. Then we don’t output and code it, and we would put it back onto the screen, and we don’t learn about security headers either.

And so we have made the perfect recipe for cross site scripting on our very first day. And that’s why it’s still in over half of websites across the internet. And so they got taught to do it wrong. And so I’m telling them that thing that you’ve been doing muscle memory on and reinforcing every day for years, [00:19:00] I need you to do it a bit differently from now on.

Paul Roberts, The Security Ledger: So you’ve got 7, 000 odd members in We Hack Purple now, where is the focus? Where are, what are the skills problems areas of learning that are garnering the most attention from your members?

Tanya Janca, We Hack Purple: Okay, so the number one class that people sign up for is AppSec Foundations Level 1, because it just explains what the heck is AppSec, how do you build a super basic program, how do you make people get on board for that it explains what every type of tool is, and all the popular types of AppSec activities you might do, so we have hundreds and hundreds of people probably thousands at this point and we have it in the Academy too, and we have Like around 5000 more people over there.

And so that’s the number one thing where most people start. But we have a lot of people. So we have all these different areas where we can discuss and we help each other. If that makes [00:20:00] sense. And we get a lot of more advanced questions in the ask me anything areas where. We all help each other. We have this big volunteer team.

And so they’re really good at, cause they’re all AppSec pros. They come in and they just like, Oh yeah, you’re having problems with that. This is what we do at my office. Oh yeah. I know this guy did this. But I would say a lot of us. So our events lately are focusing a lot on different OWASP projects.

Paul Roberts, The Security Ledger: Uh huh.

Tanya Janca, We Hack Purple: Because those are free tools that we can use.

We’ve had a lot of vendors come in and give like in depth, like technical demonstrations. So I don’t know about you, but. Whenever I’m tasked to go buy a new tool, I don’t want to sit down with 20 marketing teams or sales teams. That sounds like a very amazing piece of hell for me. And instead, what I want to see is I want to have the super nerdy engineer.

Give me a demo of exactly what it does and tell me, Oh, my gosh, you should see this. This is so cool. Watch this. That’s what I want. So we’ve been doing actually a lot of those and I remember I [00:21:00] asked the community. I’m like, do you want to see this stuff? And they said, yeah, as long as it’s like, we can ask all the in depth technical questions and it’s not a salesperson.

I’m like, yeah, no, no one. I know that salespeople are necessary and I appreciate them and that they do their jobs. But when the nerd that’s going to use it every day asks questions, we ask really in depth questions. And the salespeople are usually like, “Oh I’ll get back to you!”

Paul Roberts, The Security Ledger: So we’ve had a lot of incidents recently pen tests of. Prominent applications. There have been a couple, a few really looking at like auto telematics applications and things like this. And they all follow a very standard kind of format that the tester is using, right?

So they’re observing the application. They’re doing packet capture and sniffing the application. They’re finding all kinds of just clues vulnerable [00:22:00] APIs and hard coded credentials being sent back and forth. And, it’s just it’s just a mess what they’re able to accomplish.

In the, case of some of the telematics stuff, they were actually able to get access to the Git repositories, the private Git repositories of some of these major auto manufacturers, get onto their development slacks. And it was just like, oh my God. And it just makes you feel like, wow, these are very wealthy companies, Fortune 500 companies, manufacturers.

And yet when we peel back the covers there, AppSec is just atrocious and they’ve got endpoints that are like, weigh 3, 000 pounds and drive at 70 or 80 miles an hour. Like these are serious potentially cyber physical consequences to some of these vulnerabilities. It, makes you wonder what’s going on out there in the myriad sausage factories that are [00:23:00] producing all this software, both just.

Software and services, but also all the connected smart IOT stuff that’s populating our, homes and businesses and communities. What’s your sense on that? It seems like we got a lot of work to do.

Tanya Janca, We Hack Purple: Oh my gosh, I am so not concerned about being unemployed in the near future. There are, there’s a lot of security work that needs to get done.

Paul Roberts, The Security Ledger: Yeah

Tanya Janca, We Hack Purple: I would say that, so sometimes I sound like a broken record, so I apologize. I’d say that a secure system development life cycle is usually the thing that’s missing.

So they might have a pen tester come in once in a while and be like pew, And they find some vulnerabilities and then maybe they fix a small handful of them. But when you do a secure system development life cycle, that means we’ve added one or more security steps. Usually, hopefully, fingers [00:24:00] crossed at least one per phase.

So the. The original system development life cycle is just requirements, design, coding, testing, release. But whether you do DevOps, you do Agile, you do Waterfall, or you do something else, all of them, you still need to design and plan the thing you’re building. You still need to have requirements that you actually know what you’re supposed to do, like specifically build, not just how to build it.

There’s still that really super fun part where you code it and testing, et cetera. And so no matter what. I would like to see some requirements from the security team that are apart. So if it’s IOT, I’m going to have different requirements than if it’s an API or serverless app based on the technology, but then also. What is the risk? So is this thing, for instance, I have this friend and he designs robots that test very intense radio frequencies and I was like, what if one of your robots killed someone? And he’s that can never happen. Or my [00:25:00] entire industry would be endangered. And I’m like, so that is, would you say an absolutely catastrophic risk?

And he’s yes, for us, that is. And so then we threat modeled around that. Like with that being the core thing that we must absolutely mitigate every single potential threat for, but then other things, sometimes That could happen but it’s highly unlikely.

And if it did, it wouldn’t hurt us that bad. So maybe we’ll just accept that is a risk, or maybe we’ll turn on a security header to lower the risk of that. If it does happen, but it won’t be nearly as bad. And so it’s about you don’t want to spend 10 million dollars securing something that’s worth 1 million dollars, unless there’s lives at stake or the security of your nation.

And so I feel if you can figure out what the things are, that are super important that you need to protect and how important it is to protect them and then. Like just basically build safeguards [00:26:00] into your building process. And after you release to continue to ensure it’s secure. And that’s, what’s missing.

I wasn’t taught that in school. I had to learn it like in the Canadian government. I had to learn it at a million AppSec conferences that I’ve attended, a million OWASP events and my, work at Microsoft, the people who basically wrote the first book on a secure SDLC, I was like, yeah, I’m going to go work there.

Paul Roberts, The Security Ledger: You actually anticipated my next question too, which is the other, thing we’re hearing a lot about is attacks on supply chain adversaries, APTs advanced persistent threat actors have clued into the fact that development supply chains are extremely vulnerable and that it’s as Katie Moussouris said hack, hack once, pwn many that you can compromise a single supplier and get access to all of that supplier’s customer environments, potentially if you’re, successful,[00:27:00] I feel like secure development is actually a big part of the conversation around secure supply chain So I’m interested in kind of your thoughts

on just the software supply chain all the attacks we’ve seen including most recently this 3 CX voice over IP compromise just what you think the role of secure development, application security is in achieving secure supply chains.

Tanya Janca, We Hack Purple: I have so many thoughts. First of all, verifying that whether or not your components are known to be vulnerable, I think is really important. And so there’s tools out there called software composition analysis that can do that

I would also say that so in my opinion, the software supply chain. Is a lot more than just the third party components that you put in your app. It’s also your CICD pipeline because you need that to build your actual app. And so I’ve worked with companies where they have this on premises. [00:28:00] Jenkins, and I’m not picking on Jenkins kicks, but that’s why I picked it. But they’ll have it on prem installed and do you have backups and redundant backups?

Do you have two, like an on site backup and then a geographically distributed backup somewhere in the cloud? And they’re like, no, why? I’m like, it took you months to perfect this CICD and you have hundreds of them. Do you understand how much it would cost? If you lost your CICD, like all those beautiful tuning and perfections and tiny little tests that you’ve done, that would take forever.

So when I explained to them this is actually almost as valuable as the app itself. I was like, Oh my gosh, that’s a good point. Like we are doing backups now. And who has access to these things? Someone I know hired a co op student and they had given them permission to use the Kubernetes cluster.

And they accidentally added one too many zeros for the containers that they were releasing and released a thousand instead of 100. And [00:29:00] he’s so I have a 30, 000 cloud bill that I need to, because we didn’t know until the bill came in. And he’s and so I am having discussions. I am in the hottest water ever. And so like. Who has

Paul Roberts, The Security Ledger: That is the most expensive

Tanya Janca, We Hack Purple: to touch, yes, Yes. And so who has permissions to touch your supply chain and change it? I also think the industry as a whole, we need to start actually being a bit more responsible and being good. I would say open source citizens and we need to start submitting bug fixes and doing security on these packages.

A really wonderful, positive example that I would like everyone else to follow would be Shopify, a Canadian company where so you might not be aware, but Shopify is the biggest, the 3rd biggest realtor on the Internet. And they do this by building little tiny shops for everyone under the sun. We hack purple used to have a [00:30:00] Shopify shop where you could buy cute.

We hack purple t shirts and and hoodies obviously. But basically they use Ruby on Rails and they have a bug bounty and they have a huge app sec team that are super amazing and what they do is they fix the Ruby Gems. And then they submit them back to the entire world to be able to use. And so they have put millions of dollars into securing the Ruby framework, and that’s amazing, and that’s them giving back to the rest of the entire industry.

There are big, huge companies that use…like there’s a company I was talking to and they use an open source dynamic scanner that I won’t name but they run 20,000. Scans. A. Day! And they’ve never donated to the project.

They’ve never, and that project has like a bug bounty and there’s all these other things you could do and it, and they’ve given nothing back and legally they don’t have to, [00:31:00] but from a responsibility to our industry, I like, I would love to see like donations to these projects. In time or cash especially time if your security engineers can fit just just run scanners on some of these libraries for, Java for.

Not everyone can be dot net where it’s owned by a corporation. They work their buns off at Microsoft to keep that secure. But there’s so many frameworks that are open source and they don’t have little angels like the Shopify AppSec team taking care of it. And so we need more maybe open source angels.

Is that a phrase I could coin? Yeah.

Paul Roberts, The Security Ledger: hashtag,

Tanya Janca, We Hack Purple: that give back.

Paul Roberts, The Security Ledger: They call it the tragedy of the commons, right? That is the concept, right? Where people just use up the public free resource until it is exhausted. And I think we, I think you

Tanya Janca, We Hack Purple: Yes.

Paul Roberts, The Security Ledger: see that in open source.

Tanya Janca, We Hack Purple: Someone near [00:32:00] here has an orchard and they put up a sign that said, if you want, you can take some of our apples. And what happened was is at first, like a bunch of neighbors would take 3, 4, 5 apples and put in their lunch or something. But someone came with a truck and a ladder and just stripped the entire apple tree naked.

And it’s what are you, thinking when you do that? ’cause what you’re thinking is, I’m gonna take from all my neighbors.

Paul Roberts, The Security Ledger: So little sociopath honeypot there in the in the apple orchard. Hey, Tanya Janca, Thank you so much for coming on and talking to us and we’ll look forward to talking to you again.

Tanya Janca, We Hack Purple: So thank you so much for having me, Paul. This has been a blast.

​