In this episode of the Security Ledger Podcast, Paul speaks with Jill Moné-Corallo, the Director of Product Security Engineering Response at GitHub. Jill talks about her journey from a college stint working at Apple’s Genius bar, to the information security space – first at product security at Apple and now at GitHub, a massive development platform that is increasingly in the crosshairs of sophisticated cyber criminals and nation-state actors.

[MP3] [Transcript]

---

Innovation in the cybersecurity industry often starts with the bad guys. Hard as it is to admit, information security firms are often playing catch up with cyber criminals and nation state actors: adjusting their tools and methods to respond to changes in attacks and compromises. 

We’re seeing that dynamic play out these days in the increasing attention and urgency around attacks on software supply chains, as malicious actors have realized that they can bypass network defenses by insinuating themselves into the software and services that target organizations rely on. 

Want To Prevent Another SolarWinds? Start With Developers

Growing threats to open source platforms

Attacks on open source projects and platforms are part of that trend. Malicious actors are increasingly targeting development platforms — planting malicious modules on platforms like Github, NPM and PyPi that imitate popular and then waiting for unsuspecting developers to download and integrate their malicious code with legitimate applications. Recent months have seen large scale attacks involving scores ore even hundreds of malicious modules designed to steal data or provide remote access to environments on which the tainted applications are deployed. 

But the shift left has also put open source platforms in the cross hairs of attackers, as they look for ways to leverage weaknesses to facilitate attacks or avoid detection. As we go to print, for example, the development tool CircleCI urged developers to change “any and all” secrets stored on their system after a compromise that may have resulted in the theft of developer secrets stored in environment variables or in contexts.

Episode 216: Signed, Sealed and Delivered: The Future of Supply Chain Security

Organizing chaos

With attacks like that on the increase, how are the platforms responding? In this episode we speak with someone who knows: Jill Moné Corrallo, the Director of Product Security Engineering Response at GitHub – a position that gives her responsibility for GitHub’s product security incident response and bug bounty teams. 

Spotlight: How Secrets Sprawl Undermines Software Supply Chain Security

In this interview, Jill and I talk about the challenges of managing security on a massive and fast evolving platform like GitHub, which is used by more than 90 million developers and is a major target for sophisticated cyber actors. 

To start out our conversation, I asked Jill to tell us a bit about her responsibilities and also about how she got started in the information security space, from her start working at Apple’s Genius Bar. 

Check out the podcast using the player (above). Or, use the button below to download an MP3 of the episode!

---

Transcript

Jill Mone Carallo, GitHub: [00:00:00] Jill Monet Corralo. I’m the Director of Product Security Engineering Response at GitHub.

Paul Roberts, Security Ledger: Jill, welcome to Security Ledger Podcast.

Jill Mone Carallo, GitHub: Thank you so much for having me.

Paul Roberts, Security Ledger: it’s great to have you. Talk about director of product security engineering response. That’s a really great title and I think an increasingly important one. But for our listeners who might not of know what that entails could you tell tell them what that job is at GitHub?

Jill Mone Carallo, GitHub: Yeah, so I like to joke and say it’s the organizer of chaos. But what it is I manage the product security incident response team as well as our bug bounty team, and we are currently building up something called product security incident response engineering, which has a little bit more technical focus in our incident response.

Paul Roberts, Security Ledger: It’s always really interesting talking to security professionals like yourself, because often they’re. Kind of journey to InfoSec is, very many different paths into InfoSec, but could you give us a sense of your like, origin story and how you g got into the information [00:01:00] security field?

Jill Mone Carallo, GitHub: Yeah, absolutely. I have a unique path as many do. I originally went to college for pre-dental and thought I was going to be a orthodontist. And from there I started working at the Apple Store as a genius and went ahead and found a love for computers and really something that I had always loved and Went ahead and changed my major into computer science.

And then was approached by the dean at my school to join the cybersecurity program, which they were just building out. They saw that I had a knack for finding needles in a haystack. So from there I was still working at the Apple store. Got an opportunity to go ahead and work out in corporate on a career internship with them, I did some quality assurance testing, and then I landed a job on the product security incident response team at Apple.

I was there for about five years, [00:02:00] total 10 at Apple, and then I joined GitHub back in May, 2020 and have been growing the P-SIRT and bug bounty since then.

Paul Roberts, Security Ledger: So that is a really interesting path, just working at the Genius Bar an interesting entree into computers and in information technology. Really from the kind of end user perspective, what do you take away from the Genius Bar? Obviously you’re a problem solver, right? So that’s one thing you took away. Like I really like working on these problems.

Jill Mone Carallo, GitHub: Yeah. So problem solving empathy and transparency. It’s really huge things and it’s definitely something I’ve brought into my current role as well is understanding that there’s a human on the other side of the screen. And that’s huge, especially in incident response. Every time we have an incident, it breaks our level of trust with users.

So what our job as incident responders is, To communicate and repair that relationship. [00:03:00] So it’s really important and my time and support or the genus bar really showed me that.

Paul Roberts, Security Ledger: You manage the GitHub Bug Bounty program, and this is a very robust program. I think you 2022 was like 2 million dollars in, in bounties awarded to researchers. Really impressive. And give us a little bit of the history of that program and also tell us what’s in scope for the GitHub Bounty program.

Because every company it’s a little bit different kind of what they’re interested in and what they want to hear about.

Jill Mone Carallo, GitHub: Yeah, so the Bounty program started in 2016, and I wasn’t here for the initial of that, but I have definitely taken it now on its current journey. But from there we started as homegrown. And so we were receiving reports through bounty@github.com, managing through GitHub because we liked to run GitHub off of GitHub and receiving reports and posting stats on our bounty.github.com site. And really just learning the ins and outs. And then we [00:04:00] transitioned into utilizing the Hacker One platform. And that’s where we currently are, is managing and receiving reports still through Bounty at GitHub, but also primarily Hacker One. And so what makes us unique to this day is that we have a homegrown team specific to triaging our own reports. So all of the triage and validation is done through our team. And from there we communicate all the ins and outs in between, from the researcher to the engineer, and back and forth until we get a resolution.

Now in terms of scope we’re actually also a little unique where we have pretty much free fields. Our, all of our products are pretty much in scope. Where we tend to see areas is of being out of scope, air bypasses of the community and safety features. Some edge cases there may not result in a bounty reward or spammy or [00:05:00] malicious behavior.

And so with that, they’re also scenarios with rules of engagement. If you’re impacting users or going a little too far into malicious activity, we may steer you away from that.

Paul Roberts, Security Ledger: Yeah what’s really interesting about GitHub is, all. , many companies have bug bounty programs, but often one of the big challenges is attracting people to your platform. Who ha have the skills and talent. But for company like GitHub, you are a platform that is used by millions of developers.

I don’t know how many daily users are on the

Jill Mone Carallo, GitHub: 94 million developers to be exact…

Paul Roberts, Security Ledger: And if I had to guess, that leads to a huge volume of, folks who are finding issues or discovering problems with the platform that you guys need to manage.

Jill Mone Carallo, GitHub: It does. And what also adds to that is we’re constantly developing new features and products. And so our scope is usually growing alongside [00:06:00] that. So we see, tens of hundreds of reports coming in, but we’re able to sift through and show our new features and work with our researchers to get eyes on those.

Paul Roberts, Security Ledger: Other interesting thing is your platform is a host for other people’s code, so I’m guessing that. The vulnerabilities that might be discovered in code that is hosted on GitHub, are not in the purview of GitHub’s Bug Bounty program. It’s really focused on your products, your technology.

Jill Mone Carallo, GitHub: Correct.

Paul Roberts, Security Ledger: Okay. And what about all the code that’s hosted onto your platform? Is there any features to lead them in the right direction of, hey, you might wanna offer a bug bounty program for your stuff as well?

Jill Mone Carallo, GitHub: We have a lot of education resources and we have our GitHub security lab, which I’m not all too involved in, but they are a sister team. And they have their bug bounty where they interact with more of that open source community to encourage opening up the fields to our open source partners to maybe [00:07:00] run a bounty or we recently re or we recently released a new feature private vulnerability reporting. And so what that is we’re able, we’re opening up free on the platform for anybody to send a private report of a vulnerability in someone’s code, so therefore making a big impact to the community.

Paul Roberts, Security Ledger: Yeah, so that sort of streamlines the process cuz one of the big problems that people have is, I found some vulnerability in this open source library, but what do I do with it? Who do I go, I can, I guess I can, email or message the developer, but how do I run it up the flagpole?

And I guess that’s what you’re, that’s what you’re facilitating there.

Jill Mone Carallo, GitHub: Yeah. Because we’re home to so many developers, security isn’t just an opportunity, but it’s a responsibility and our work greatly impacts the security, and we see that daily.

Paul Roberts, Security Ledger: So what can you tell me about what’s what’s hot and happening in the vulnerability space at, at GitHub? I We’re used to the OAT 10, in some ways things. Don’t [00:08:00] change that much. Cross-site scripting, SQL injection, those types of vulnerabilities tend to be very common.

Is that most of what or are you seeing changes in, in what gets reported and the problems that are cropping up?

Jill Mone Carallo, GitHub: We’re pretty aligned with the rest of the community. We have been told that we are a harder target at times, which makes me feel good that, maybe we’re doing something right there. But we tend to see pretty much what everyone else sees.

Paul Roberts, Security Ledger: we’re seeing a lot more about supply chain threats and attacks, right? So this is platforms like GitHub or NPM or Ruby Gems or, gi, GitLab increasingly interesting to malicious actors, right? They’re figured out like, Hey, if I can. , put a bogus package on there and somebody pulls it into their project, then suddenly I’m I’ve got access to a lot of their customers.

Have, has that, have you seen that filtered through in the work that you do? Either in the bounty side or the other work you do? The incident response work you do? The, just the increase in activity there on the supply chain.[00:09:00]

Jill Mone Carallo, GitHub: Yeah. So obviously it’s the anniversary of Log4Shell or Log4j we’re all still licking our wounds from that. But what we’re seeing now is with the supply chain that developers are really missing the mark in. Older vulnerabilities in the CI/CD pipeline. Things where we’re not necessarily focusing on what’s old and what’s been around, but we are seeing things where they’re new and fresh and shiny and exciting because everyone’s pushing for that new feature to get ahead of one another.

Always love to tell our engineers. Slow down to speed up usually starting from features that, maybe haven’t had a little bit of love in a minute. In 2016 we saw that Alva from our security lab presented on the exact scenario from Log4j and so looking at that scenario were able to pivot off that and show our engineers. There’s all these talks going on at Black Hat and Defcon and BSides and whatever. Let’s look at those. Let’s [00:10:00] peer around the corner. Let’s make sure we’re really looking at those scary cobwebs in the back of the closet and addressing things from all ends.

Paul Roberts, Security Ledger: Yeah. Log4j was everybody’s worst nightmare, right? Something that’s everywhere. Been around for a long time and highly vulnerable. One of the big sort of supply chain attack vectors that, impacts platforms like, GitHub are things like like typo squatting attacks, right?

Where people, people will put malicious packages, libraries that just are, have a similar name, just typo squatting on the web, right? Like domain squatting, right? And it works strangely well from a platform perspective, is there really anything you can do about that in terms of monitoring and detection of huh, we’re seeing these new projects pop up that have names that are pretty similar to these really popular projects and, just some intelligence, some heuristics to detect that type of activity.

Jill Mone Carallo, GitHub: Yeah, it’s a hard [00:11:00] line because you want to make sure that your user is able to take advantage of changing their name of a project or their username or anything like that, but still protecting the community at large. Specifically for us, where we look is requiring a 90 day wait period on, say, a regular, non-popular package. So after that 90 days, we’re able to release that username or, the package name or whatever the case may be, and then that can be opened up to the community for popular packages where you have a lot of folks using those packages. We actually immediately tombstone that name to protect everyone in the supply chain and then anybody using GitHub actions and those names, those will actually also be immediately deleted as well. So we try and balance user expectation, but safety and security too.

Paul Roberts, Security Ledger: We’re seeing also a [00:12:00] lot of I know Checkmarx had a report out today on a big supply chain attack. That was clearly automated. I think it was like a hundred thousand packages or something like that with, know, Phishing links in them. We’re used to like, in the context of Twitter, talking about automated inauthentic behavior or stuff like that.

Is there, are there ways as a, open source repository platform to detect that type of automated behavior that might correlate with some kind of, malicious campaign? Or again, is it , too noisy. We’re gonna interfere with too much legitimate activity in our quest to, to find that one, that needle in the haystack, that’s the malicious campaign.

Jill Mone Carallo, GitHub: Yeah, I haven’t read that article, so I can’t speak too much on that. But from just a more broader general tip, it is. Hard to focus in because it is so unique. Every situation, you can put in rules, you can look for that behavior, typically I would lean towards how many issues are being [00:13:00] created or PRS being created in this space and is there common string, that also can. That can be misleading as well and can be actionable, regular action force at user. So it is a complicated space to, to focus in on.

Paul Roberts, Security Ledger: Okay. So you oversaw I think basically like a hundred percent growth in the amount of bounties paid out on GitHub’s Bounty program, I think between 2020 and 2021 so what does it take to grow a Bug bounty program and make it more successful and interesting is it just, we’re fortunate to be a platform that’s gaining users and therefore gaining attention and that, that just spills over into the Bounty program, or are there things you do concretely to increase engagement, increase involvement dangle more money before people, I don’t know what the tricks are.

Jill Mone Carallo, GitHub: Yeah, money always helps, but it’s a sprinkling of everything. Obviously having a popular platform helps but also ensuring that [00:14:00] our researchers are happy and heard and feel like they’re connected to us. One of the things that we did in the last year specifically is we hosted our first solo live hacking.

Years previous, we had been part of the H1702 multi customer event. But this year we decided to take that on solo and it was a bit of a chaotic few days of that event, but we got a lot of. Refreshed energy on looking at our platform and being able to meet up with our researchers in person in Austin was really fantastic because I always say breaking bread with someone completely changes your communication, and especially with our researchers, we saw the impact of that because, We had one researcher who was in the other room and we were in the triage room and we kept messaging through the platform. But finally I was like, you know what? Let’s just walk over there, have a real conversation. He’s here. Let’s take advantage of it.

Paul Roberts, Security Ledger: Yeah. Right.

Jill Mone Carallo, GitHub: [00:15:00] And afterwards he approached me and he was like, thank you for coming and talking to me. It was so meaningful to have that conversation and see that you genuinely care. And we’re trying to understand versus that perception that just because there’s a person on the other end of the screen and you can only say so much without writing a novel of an email back, you know that they do care.

Paul Roberts, Security Ledger: Yeah, particularly come coming out of the pandemic as we all are. We’ve all been accustomed to just of being like we’ll just, let’s just get on a Zoom or Google meet and, but there is you do lose a lot. You absolutely lose a lot.

Jill Mone Carallo, GitHub: And it’s important that we are attending conferences now as things open up and that we’re reaching out for feedback from our researchers. We’re always asking, what can we be doing better? We have some fun and exciting things in the new year coming out that will be publicly announced on our blog and just really excited for those to come out.

And that was directly from feedback from our research.

Paul Roberts, Security Ledger: Any any hints on what [00:16:00] those things are?

Jill Mone Carallo, GitHub: Might be the T-shirt that I’m wearing.

Paul Roberts, Security Ledger: It’s interesting. In some ways it’s almost like a retail business, right? Like, how do we get people, how do we get talent? How do we attract the the, critical mass of talent to our bounty program? And that’s a marketing problem in some ways, even though what they’re doing is like highly technical, right?

Jill Mone Carallo, GitHub: Yeah, and diversifying the incentives. There is this thought that monetary value is the only thing that’s important, but there’s reputational. There is non-monetary rewards, such as things as swag and recognition and things like that.

Paul Roberts, Security Ledger: True. The security space is still, very focused on, traditional vulnerability, discovery and patching and stuff. That’s where most of the products are. That’s where most of the investment is. But this supply chain issue is getting bigger and bigger. Solar Winds was the, the test case there. Log4j as you said, underscored a lot of the open source risks that’s out there. Where do you see that discussion going and what do you, I think a lot of people look to [00:17:00] the platform providers themselves, again, you guys, npm, and, some of the other platforms as key players in this.

Where do you think things are gonna go in the next year in in, in, in terms of trying to address some of that supply chain risk?

Jill Mone Carallo, GitHub: Yeah, so there’s a few areas where we can address that. Specifically Mike Hanley, our chief security officer, recently attended the White House Open Source Security Summit recently being also a year ago, I think at this point, . And from there we came out of that. With the desire for tooling and education and really helping our community in that way, in the open source community and just making sure that everyone has the ability to have a baseline of security. And I think that’s super important. And these conversations such as Mike attending the summit really enforce that because it gets the community talking and aligning. And that’s the biggest thing that we can [00:18:00] do because we’re all solving the same problem in that workspace. Why not put our heads together?

Paul Roberts, Security Ledger: Yeah. Do you think. Bounties are really bug bounty programs came about, I don’t know, yeah, 17 years ago, and over that time they’re not, they haven’t changed that much. It’s basically the same model. ZDI, was one of the first, not the first, but one of the first. And it’s still pretty much the same. Obviously we’ve seen platforms, you Bug Crowd and HackerOne and so on come along that, that facilitate a lot of the operation of these.

Do you think the bug bounty model is still working well? Or do you think that it needs to change and evolve to suit? The evolving, development environment, development world.

Jill Mone Carallo, GitHub: I think it’s working well for where we are right now. I think as we further technologically, obviously there will be advancements in that, but I think where most programs can seek to [00:19:00] expand is to spend more time with their researchers. At the end of the day, we wouldn’t have bug bounties if we didn’t have people. And and also realizing that their work is incredibly valuable. It is time, money, and effort that they’re spending to report something and to understand and take that seriously and give it a good college try before just chewing it off. I think is truly important. I think as we grow keeping researchers engaged gets harder.

As you start to see less reports come in, the engagement piece is what most people have to work on. So for us we’re always looking to expand. How do we get rejuvenated energy into the program

Paul Roberts, Security Ledger: Right.

Jill Mone Carallo, GitHub: We have a whole team dedicated to it, and we want them to fill their time in and be working in the space that they love.

Paul Roberts, Security Ledger: Final question, for our listeners, I’m always [00:20:00] interested what, if they might be interested in getting involved in this, doing bug bounty research for, GitHub or others. What’s your advice to them as to first steps, how to of get, get involved in that?

Jill Mone Carallo, GitHub: Just ship it honestly. , we see a lot of hesitation in folks who will approach us in different means, Twitter or various other ways, or come up to us at a event or conference, and there’s a lot of hesitation. Because it’s something scary and new for some folks, and we love encouraging folks to submit their first bounty to us.

We always enjoy seeing a new name pop up. We’ll do our best to coach them along if they’re on the right path. We’ll send them some documentation, help them out as much as we can because we want you to be successful. You want, we want to see that relationship grow. It’s really great for us on the other side of the screen to watch your development as a person in this space.[00:21:00]

Paul Roberts, Security Ledger: And just to entice them, what’s the biggest bug bounty you can get from from GitHub?

Jill Mone Carallo, GitHub: We paid our highest bounty last year, which was $50,000. But we have a whole slew of everything in between from there.

Paul Roberts, Security Ledger: Down to t-shirts!

Jill Mone Carallo, GitHub: T-shirts, maybe some patches, some other things. Those

Paul Roberts, Security Ledger: Final question. If folks are listening and they’re working at a company that, particularly, outside of the, so outside of the high tech industry, right? A lot of these kind of older economy companies that make stuff, machinery, but now it’s Internet connected.

 They might know they need something like this, but not have the wherewithal to get going with it. So for the companies any advice or thoughts on standing up one of these bug bounty programs for your products and getting it launched and getting people to come and help you?

Jill Mone Carallo, GitHub: It’s incredibly valuable because fresh eyes are always the best eyes especially when you are so entrenched in what you know and care about. So I say that for everything from, submitting your first report all the way through to [00:22:00] your millionth report or addressing a new feature. You always wanna bring somebody new in to make sure that you are addressing something you just might overlook?

Paul Roberts, Security Ledger: Jill Monnet Carlo get home. Thank you so much for coming on and speaking to us on the Security Ledger podcast. It’s really been a pleasure having you

Jill Mone Carallo, GitHub: Thank you so much for having me. It was great.