Podcast: Play in new window | Download (Duration: 42:42 — 39.1MB) | Embed
In this episode of the podcast (Episode 252), host Paul Roberts speaks with Colin O’Flynn, CTO and founder of the firm NewAE about his work to patch shoddy software on his home’s electric oven – and the bigger questions about owners rights to fix, tinker with or replace the software that powers their connected stuff.
[Video Podcast] | [MP3] | [Transcript]
There is no shortage of interesting talks at this year’s Black Hat Briefings in Las Vegas – with everything from an address by the U.S.’s acting National Cyber Director Kemba Walden, to a string of talks on supply chain security (thanks GitHub!) to an excellent discussion of the pros and cons of software “lemon laws.” (I wrote a post on “must see” Black Hat sessions which you can check out over on ReversingLabs blog!)
But Black Hat’s most radical and norm-busting talk isn’t about any of those things. Instead, it’s about how a fixture of our everyday lives – our ovens – may be lying to us, and the implications of that for owners of software-driven stuff everywhere.
I’m speaking of course about Oven Repair (The Hardware Hacking Way), a presentation by Colin O’Flynn, a Canadian security researcher and CTO of the embedded software security firm NewAE. Colin’s talk takes place Thursday, August 10 at 10:20 and if you’re attending Black Hat, you should make it a point to attend. You can also read about Colin’s research over at his website, colinoflynn.com.
Going cold turkey…
I had the pleasure to interview Colin on The Security Ledger podcast about the problem he encountered soon after he and his wife purchased their home: a Samsung electric oven that was routinely telling them their oven was hotter than it really was, resulting in woefully undercooked food including a turkey that needed more than an hour of extra cooking time.
O’Flynn wasn’t alone. Going online to research the problem, he soon discovered that many other Samsung oven users had encountered the same problem across a range of ovens. Even his in-laws were finding that their Samsung oven didn’t accurately display the temperature.
Encore Podcast: Chris Valasek on Hacking The Jeep Cherokee
For most home appliance owners, that’s a problem that would beget calls to an appliance repair shop, some ‘shot in the dark’ efforts to recalibrate the oven or (pricey) repairs to replace the “usual suspects” – balky temperature sensors or heating elements. Many owners would balk at even trying those fixes, sending 100 lbs of metal, plastic and electronic waste to a landfill.
Reversing the oven
But Colin’s not most appliance owners. He has a Ph.D in Electrical Engineering with specialization in things like hardware reverse engineering, penetration testing and the like. Looking at his oven, Colin concluded that the problem was almost certainly software-based rather than hardware related, and he set out to fix it.
What followed was a fascinating quest to get to the bottom of his wonky electric oven, including the use of side channel cryptographic attacks on the oven’s microcontroller to circumvent security features that deny customers (or, really, anyone) access to the software that runs the device.
Eventually, he was able to extract the underlying embedded code and apply a small, binary patch to the firmware. His patched fixed a badly designed temperature sensing feature – likely coded by Samsung – so that the oven display correctly reflected the actual temperature of the oven. And it allowed the oven to keep its temperature at the desired level, resulting in better culinary outcomes.
Do software locks enable security… or secrecy?
But the issues that Colin’s research exposed are far bigger than a balky temperature sensing feature. In fact, they’re fundamental questions for our fast-evolving Internet of Things. Among them: how the primacy of laws like the Digital Millennium Copyright Act (DMCA) is helping to conceal the ‘elephant in the living room.’ Namely: the proliferation of poorly written, poorly managed software and hardware in life- and safety critical devices that is seeding homes, businesses and communities with unreliable but opaque devices.
Episode 241: If Its Smart, Its Vulnerable a Conversation with Mikko Hyppönen
For more than a quarter century, the DMCA has imposed draconian restrictions on electronics owners – preventing them from disabling software locks and other DRM (digital rights management) features with the threat of a federal crime. As I pointed out in my recent Congressional Testimony, that has led to the proliferation of software locks across all manner of devices and for purposes far afield from preventing digital piracy of movies, music and video games – the ostensible purpose of the DMCA.
A trip inside the embedded software sausage factory
Colin takes us inside the software “sausage factory,” so to speak: uncovering insights into the ways that manufacturers’ choices about both hardware and software interact in ways that can impact device performance and longevity (not to mention security) for the better – or worse. His work on his oven – which was more than a decade old and not “smart” – suggests how greater owner access to the workings of our appliances and other connected stuff can lead to more transparency, scrutiny of lax development and security practices and better outcomes for consumers, communities and the environment.
You can check out our podcast using the player above, or view a video of my talk with Colin or read a transcript of our conversation below.
Video Podcast
Transcript
[00:00:00]
Paul Roberts: Colin O’Flynn, welcome to the Security Ledger podcast. It’s great to have you on!
Colin O’Flynn: Thanks so much for having me here
Paul Roberts: Yeah, it’s great. So I reach out to you because you’re going to be a speaker at next month’s Black Hat briefings in Las Vegas, and you’re going to be presenting a kind of updated version of a presentation that you gave on this really interesting hack you did of your oven, to solve a really knotty problem that you had identified with the software that runs this device. So we’re going to talk about that. But before we do that, Colin, tell our viewers slash listeners just a little bit about [00:01:00] yourself and the, company you founded NewAE technology
Colin O’Flynn: For sure. I started my background maybe at the beginning to all electrical engineering really so I know you know, it’s security is a great space because you get people from all over i’m terrible on network security, for example, but my real love is electrical engineering. I started on the other side doing wireless protocols, ZigBee IP and all the sort of IOT stuff.
Honestly, when it was called wireless sensor networks and it was just a cool research domain, it seemed that spun me into security. I saw a talk by someone, Paul Kocher, that was part of this, guy that, yeah, so he, really started this field of side channel analysis where it’s, or side channel power analysis, I should say and even some timing and attack work, of course he did, but right, it was like.
It’s amazing. Like devices are leaking information, so they run a cryptographic protocol, and it seems like it shouldn’t work. But just by observing the power used by the device on very, fine [00:02:00] scales, you can see differing number of bits going across the bus and stuff like that. So that is really what It’s pivoted me, to be honest, as part of that, I did research in the area.
So I did a PhD in it and built tools that I was using for my research. And the tools are all about taking measurements effectively doing it with open source. And the point of that is that for researchers, it’s really nice to have reproducible tools. Coming from the engineering side, I also know, lots of engineers.
That would say oh, these attacks are important. You need a PhD to do them. You need all this equipment which wasn’t true. And the point of the company I started a new way of technology was to make reproducible equipment that researchers could use, but also engineers could use to interface with researchers.
And I guess it it turned out it’s like everything in life, a lot of luck in a way that it was right at the time. There is more interest in hardware security. So we did Kickstarter on this Chip Whisperer Light, which was one of the implementations with the hardware. [00:03:00] And yeah, since then we’ve grown, so there’s about 6 people full time here in Canada, 5 of us in Halifax 1 in Ottawa.
Paul Roberts: So Chip Whisperer talk about what it is. Is it hardware? Is it software? Is it both?
Colin O’Flynn: It’s a bit of both. The original idea of it was this full platform. So if we’re doing side channel attacks and fault attacks you normally the kind of setup I would always show is if you bring a paper up, it’s like the researcher has in their lab. So at the university, for example, we had a.
A really nice oscilloscope that’s worth like 200, 000, and you don’t need that oscilloscope, but it’s just what they bought on some great grant one day. And so the problem I found is that when you go to recreate stuff. The researchers obviously using the super nice oscilloscope. And so I tried to make hardware that could replace the oscilloscope so that you could have a package of, Hey, here’s the software, the hardware, everything right for doing a power or fault injection attack.
so yeah, so physically as a reference, [00:04:00] this was a later revision, but on video, it’s, there’s a little box that does the power measurement and fault injection. And then typically there’s some target that’s a reference microcontroller that runs firmware or something like that
for learning.
Paul Roberts: it costs 200, 000.
Colin O’Flynn: No, exactly. So that’s the thing. So it was the first version, we were like 250 bucks, and then we, have spread out between upper and lower. So now it goes from $50 to $4, 000, depending on variance.
Paul Roberts: And who’s, the customers for NewAE and what types of people in your experience are interested in this using, Chip Whisperer and, what types of things are they using it for?
Colin O’Flynn: so we have a split of customers. Some of them are strictly researchers in terms of classic academic right?. Using it to test a new algorithm. Are they post quantums? A good example. There’s lots of post quantum algorithm testing work testing new attacks, things like that. It’s roughly a third.
We have this third split between that a [00:05:00] third split is what we call commercial. And this is typically engineers, right? That are working at a company. And they simply say, Hey, we should recreate these attacks to test our stuff. And then a third of it tends to be more like penetration testing security companies or some government labs, which is doing are doing similar work, right? As penetration testing on internal systems.
Paul Roberts: You’ve been interested in kind of hardware engineering design, as you, you say, in your LinkedIn profile since you were like 10. As a kid, this is something that you were just fascinated with. And you have the the RadioShack hardware and all this stuff. What’s your what’s your take on embedded systems, particularly we’re going to talk about your oven work, particularly in the sort of consumer space the retail space, the stuff that’s being sold into homes and small businesses, the engineering culture that, or the engineering processes and [00:06:00] status quo that’s producing all this stuff.
Colin O’Flynn: It’s interesting because if you think of engineering in general it’s you think engineering, maybe you think of building bridges or something like that, where it’s very strict safety requirements. And in Canada, for example, especially we have the professional engineer, which you’d have liability.
If I designed a poor bridge I can go to jail because I. Designed it not to the standards. It should be. So it’s really interesting in software because we have computer engineering programs, which are supposed to have similar amounts of things taught to engineers.
But then I think what is that it’s, it doesn’t feel the same, at least to me, right? The level of sort of rigorous code review is not always applied in that way, especially on consumer appliances, because cars, for example, have a. A lot of standards around safety and what they actually have to do and some testing they have to do, and you could [00:07:00] argue if that’s effective or not, whatever, but that’s it.
But it’s there, right? But like this oven, for example, that I looked at, it has a microcontroller that’s controlling the heating element, and it’s always wired to a 240 volt, 30, 40 amp circuit. If that just turned on when I’m on vacation, what happens? Like hopefully the oven is physically capable of surviving that? I don’t know. I don’t know if that’s tested, but it’s just firmware that turns on the heating element.
Paul Roberts: And there isn’t there isn’t the same regulatory attention to that aspect of the device’s operation. And I would say the same, I think is really true with, you mentioned automobiles yes, there are safety requirements for automobiles. But I’m not sure they actually extend to the software that’s on automobiles these days because we’re seeing a lot of really scary stuff and there does not seem to be Any consequences for the automakers about
Colin O’Flynn: the
Paul Roberts: scary stuff that’s coming out. Let’s talk about your kitchen and your [00:08:00] oven because this is again going to be the focus your, some really interesting research. It’s going to be the focus of your talk at Black Hat. Of all how long have you had this oven?
What type of oven is it? And what was the problem that you became aware of with this oven?
Colin O’Flynn: So it’s an old Samsung oven. I don’t it was with the house when we got it in 2016,
and I’m sure it was 10 years older, at least based on what I’ve seen and so it has, and a lot of ovens do this, feature
where
Paul Roberts: Gas oven
Colin O’Flynn: it’s electric. Yes, electric oven. And when you turn it on it goes into preheat mode and it shows you the temperature as it’s heating up.
And then once it hits the set temperature, it just displays the set temperature and you have no idea what the actual temperature is. So you can put an oven thermometer in to, to see and we had noticed that it would go off temperature pretty heavily. So it would say it was [00:09:00] 375. It was actually at 250 and either you have a thermometer in there, or if you reset it, it’ll show the right temperature again, like you turn on and off, right?
It shows the right temperature again. And so you can see oh, this thing’s way off what it should be.
randomly
Paul Roberts: the oven off and then turn it on, then it sent, the temperature would reset to what the actual temperature inside the oven
Colin O’Flynn: exactly.
Paul Roberts: You
Colin O’Flynn: the logic seems to be basically that well, it’s below the set temperature, it’s displaying the actual temperature until it crosses over then it never updates it again, just says 375, right? So if it’s below and you reset it, then it’s going to. Realize, or not, it knows the whole time, but it’s going to show you
that it’s lying to you.
Paul Roberts: baking, that becomes a real problem.
Colin O’Flynn: Yeah. Yeah. And we really noticed it when we did a turkey one year and it took it was like, oh, this should be done, but it’s. Not even close, right? It’s an hour late now.
Paul Roberts: And you [00:10:00] got guests sitting there twiddling their thumbs at the table. Yeah.
Colin O’Flynn: just, us that
Paul Roberts: Okay.
Colin O’Flynn: we got dogs. That was the, photo in the talk is what the hell we’re going to be begging here. But yeah for a lot of people I’m sure they’ve had this frustration.
Paul Roberts: Yeah. So when you encountered this problem, did you like hop on the customer support forums and try and figure out like what the solution is or if there was a fix or, a setting you could adjust or something like that?
Colin O’Flynn: I did. Yeah. So there is, there’s a few settings you can try recalibrating it and. There’s some settings on convection that will automatically drop it. So it was none of those. And to be honest, I didn’t think too much of it until there was an article from another guy in Halifax. He, actually had an oven fire.
And then in that article, they mentioned, Oh, there’s this someone’s trying to do a class action lawsuit against Samsung for this group of ovens, including mine. And they list all these complaints that are roughly the same the temperature is way off, sometimes high, sometimes low [00:11:00] and they’re blaming the thermometer these are lawyers, I think, are just picking out an item and saying, oh the temperature sensors defective which, which to me, it seemed like it wasn’t because if you reset it, it shows
temperatures.
Paul Roberts: to be a software issue. The
Colin O’Flynn: Exactly.
Paul Roberts: Temperature is being read and and figured into the operation of the oven. So you talked to your wife and said, I need to pull the oven out from the wall for a week or two.
Colin O’Flynn: So I did. I bought a spare oven
Paul Roberts: Oven.
Colin O’Flynn: board because this seems like a bad idea
Paul Roberts: Even before we get into that was there like when you went to look into what’s going on with this oven, was there information that you could get like schematics and service manuals and stuff that helped you understand how the device worked?
Colin O’Flynn: Yeah, there was a service manual some of the sites that sell them type thing. So I was able to get a service manual. You found a number of people [00:12:00] complaining about this as well. We didn’t end up getting a repair person in because when I was looking, a lot of people said, oh, they’ll just try swapping different things out.
It’s not. It seemed to be just like unless the thermometer is wrong, but I could tell that wasn’t it the heating element could be broken, but seemed not to be.
Paul Roberts: So in your head, you were like, this is a software issue.
Colin O’Flynn: yeah. And to me it looked a lot like if you’ve worked with PID controllers before, it looked like this isn’t really tuned well, cause it’s working it’s like spiking and
Paul Roberts: Yeah.
Colin O’Flynn: going to the wrong temperature long term and stuff.
Paul Roberts: Right.
Colin O’Flynn: original. Yes, it’s like it feels software is someone that’s written bad software, right? This feels like bad software
Paul Roberts: Okay, so what was the process you used to get the software out of your oven and be able to look at it and see how it was written?
Colin O’Flynn: So the microcontroller and it’s an old [00:13:00] Toshiba 8 bit 16 bit microcontroller So the first step was see what’s the security they have
Paul Roberts: When you
Colin O’Flynn: it.
Paul Roberts: old, what are we talking?
Colin O’Flynn: I think it’s sort of 2000, that seemed to be the release. So it’s EOL now
Paul Roberts: Okay, so yeah, two decades old? OK.
Colin O’Flynn: Yeah, 2003, maybe I can’t remember exactly and this is based on press releases or something, so not hard dates, but yeah, it’s been EOL at least like four years, I suspect now, I think
Paul Roberts: Which is if your oven is circa 2010 or something 2008, that might not actually be crazy that it would have a six or seven year old microcontroller in it. But
Colin O’Flynn: exactly right.
Paul Roberts: so what were the impediments to, so you got this microcontroller, it’s on the back of the oven, you pull the oven out from the wall and you got the board there, the microcontroller on it, what do you do as a hardware engineer then to suck the software out of [00:14:00] it and be able to look at it and figure out what’s going on?
Colin O’Flynn: Yes, the first step is always just whatever research you can do. So in this case get the part number off, check the data sheet. And for this type of device, it’s all public. It’s not a secure device where you then you initially run into that. But luckily the device, it had a like an old bootloader in it that should just work to read it out. They had a few security features they enabled. So part of the talk is talking about using power analysis to recover the password. The main Toshiba, the microcontroller manufacturer, had allowed Samsung to set a password to stop someone from reading it out.
And then they had a second feature that would just disable… You could set this flag that would turn off any reader right to the chip as well. So they also had set that flag. So initially I had to bypass both of those. The other problem, of course, is it’s an old microcontroller, so you don’t just have.
Modern tools don’t work with it. So I also had to find an old and luckily I [00:15:00] found on eBay an old dev kit for this device, which then had right, like the software and a programmer, stuff like that to actually physically talk to it.
Paul Roberts: Yes
Colin O’Flynn: And
Paul Roberts: Shipped to you on a cd rom…
Colin O’Flynn: Yeah, and a little Windows XP, it only worked in Windows XP. So luckily, a little VM running. they had removed it.
Paul Roberts: to the sort of age of some of the underlying systems and you know This microcontroller might be circa 2000, but it’s clearly utilizing elements that are significantly older than that.
Colin O’Flynn: Exactly. And I’m sure it’s just the code has been compiled and I’m sure it’s the same, almost same code. As, I said, my parents had a more recent version of this oven and that version has a, I think it’s an arm based microcontroller in it. It’s a little more recent there, but it seems to have the same program flow flaw.
I suspect this is like “ovencontrol.c,” compiled for …everything.[00:16:00]
Paul Roberts: So you were able to bypass these controls, these safety locks, basically, that Samsung had put in place to prevent you from getting access to the embedded firmware. How long did that take you, and how many tries? I remember talking to Charlie Miller and Chris Valasek about their Jeep Cherokee work, and they talked about how many times they bricked end unit that they were using to try and get to the canvas and having to keep bringing it back into the dealership and be like, Oh, the radio is not working.
Colin O’Flynn: Yeah,
Paul Roberts: were you, was that the case here? Were there, were you bricking your oven or were you able to extract it without too much trouble?
Colin O’Flynn: It wasn’t too bad. So I did make the other cheat I did is I made a little. Test board. So I found because the oven control board is like 200 ish. If you have to buy a new one and I made a little board that kind of had the chip on it, right? It’s a test. So, then I could at least do it didn’t have the firmware, but I could set similar things to all [00:17:00] my testing on that.
Although it did turn out that there was when you bypass one of the features, there’s a chance you erase it. And on my little test board, it rarely happened. Yeah. And on my I bought that spare board off Ebay, it didn’t happen. But the first time I did it on the real oven, of course, it just bricked it.
So then I, and I think that was like 8 PM at night. So the repair shop locally is closed that has a part. And I was way too overconfident because my wife had actually was going to bake something that I said, Oh, let me fix. Let me put the fixed firmer on it. Let’s try this.
Paul Roberts: Yeah.
Colin O’Flynn: And then it was like, yeah, what about tomorrow?
Paul Roberts: The pressure’s
Colin O’Flynn: Yeah, so it just, erased. And then finally, when I got the replacement board, it turned out they actually, they didn’t, they only enabled one of the security features. So they, stopped enabling both of them. Which I think, so the feature that they originally had enabled would have prevented them from reading out anything.
When I got the second board, [00:18:00] and I suspect maybe they had some issue with returns and they wanted to see, hey are these getting corrupted? Like the flash firmware, is it getting corrupted or something? I think they didn’t enable the second feature so they could do that analysis is my guess anyway.
Paul Roberts: So when you took a look at this firmware from the oven what were you able to determine about how it was written? One of the things you say in your presentation is your oven is our ovens are lying to us, right? So at some level, these. Devices are being programmed to represent a certain operating state, but that may not really reflect what’s actually going on in this case in the oven.
Could you actually see that happening in, in the code that like the actual temperature was not being relayed or it was what, were [00:19:00] you able to discern about the operation of the oven just from the firmware you extracted?
Colin O’Flynn: so it was interesting. You could actually find luckily because it’s a pretty simple old code. I basically wrote a little monitor that would dump memories. To a serial port that was on the oven. And with that, I could actually see eventually find where it’s storing the firm, the temperature in memory.
So I could make these plots of here’s the actual temperature, right? And actually, when it’s turning on and off the elements. So you could see. And even without looking at the code flow you could see which, is the original question I wanted to answer was is it the temperature sensor?
Because I don’t think it is. And I
Paul Roberts: it wasn’t.
Colin O’Flynn: Yeah, it’s right. it’s reading correctly. The only thing it doesn’t. And this might have been one of the reasons they have this, cheat in there where they don’t show the varying temperature when the heater turns on. I think due to electrical noise, it’s like the temperature goes off by 15 or 20 degrees.
It’s just whenever the heaters on [00:20:00] instantly, it’s not heating. It’s just the electrical noise of that heat. I’m guessing from the wires. So I think 1 of the reasons they did this, firmware hack was actually to cover. Because if you’re a user and you see the temperature jumping around, right?
Paul Roberts: you’re going to, you’re going to say, Oh, the oven’s broken or something’s
not
working.
Colin O’Flynn: So they, if you, they say in the manual, Oh, don’t worry. That’s just electrical noise. No, one’s going to read that and believe them. It’s
Paul Roberts: talk about your, fix for your oven. So what did this require you to do to get it so that it was more accurately reflecting the temperature inside the oven itself?
Colin O’Flynn: So for that, once I had the code out of it which is really just the binary firmware, right? Then it was a sort of classic reverse engineering to, to figure out, okay, where in this binary is any of the control logic. And there was a few different things I tested. So one of them was [00:21:00] just making it display the actual temperature.
And so that at least let you knew when things were up the second fix I did, which so I’ll get to the issues with it right in a second. But the second thing I fixed was that the actual control logic, I basically made it stay in that preheat mode where it’s actually. Correctly, more aggressively controlling the temperature.
So it, it would get back up to 10. The final issue, which I haven’t fixed yet. Is there some fail safe? I think, because right now, every few days, the oven just won’t heat. It will go to the 86 Fahrenheit and that’s it. And so you have to flip the circuit breaker. You can’t just turn it on or off on the control panel anymore. reboot
Paul Roberts: kill the power.
Colin O’Flynn: Yeah. And the first time that happened, I was away. So my wife, okay. You have to reboot the oven from the circuit breaker. And I think that’ll work and it did, but yeah. So, that’s the one last thing.
Paul Roberts: I imagine some conversation with your wife standing at the circuit breaker and you’re being like, [00:22:00] “Well…try…”
Colin O’Flynn: Yeah. And we just had the panel replaced, but then they, I think that was at least maybe accurately marked, but a bunch of stuff wasn’t
Paul Roberts: yeah.
Colin O’Flynn: it’s the classic I think it’s the double one.
Paul Roberts: When you talk to people who are like reverse engineering software digging into stuff You’re finding all kinds of interesting stuff in the developer comments or you know Just indications about who wrote this when it was written things that were you know Commented out that used to be in there like any insights in your research?
into You know how this sausage got made was this is this Samsung code? Is this contractor code? How old is it? Like any of that stuff or was it pretty clean
Colin O’Flynn: It’s pretty, opaque. The thing is, because it’s this old processor, they really, I think they really tried to optimize for size.
because The only reference I have is one of the passwords is, has Samsung in it.
Paul Roberts: Okay
Colin O’Flynn: it must’ve been, and the board is marked [00:23:00] Samsung, stuff like that. At some level it is Samsung. It’s not completely with ECU is now it’s a different entire tier that’s made it whether or not they internally. Had someone else do it. I would suspect it’s probably them based on the fact it’s used in such a wide line of products.
Paul Roberts: So you added these new features, you tweak the temperature monitoring sensor functionality, right? And then you recompile the code and you basically could flash the microcontroller with the updated code, is that right?
Yeah and it ran.
Colin O’Flynn: Yeah, it ran. And, everything I’m doing, I guess I should give the caveat, right? This is all just little binary patches.
Paul Roberts: Binary patches, yeah,
Colin O’Flynn: minimal what you can do. It’s okay, don’t do this jump. So, in that, in the case of the temperature sensor update you could just see where they say, “oh, if we’re out of preheat mode only display 375 or whatever your set temp is,” right? So you just comment out that jump.[00:24:00]
Paul Roberts: And if you were a non, if you were an oven owner who was not a Ph. D. electrical engineer, is it, like, how would you go about installing patch for their Samsung oven? What would the process be? Like pull it in, pull it out from the wall, like patch into that microcontroller and get their laptop and upload the software.
Like, how is it, how would you affect that change?
Colin O’Flynn: that’s, and so I, thought originally, so the original plan, which of course when you’re like, Oh, so much time to do this was to try to make a little, cause you could run this off an Arduino or something. It’s just a serial protocol is all you need. Exactly. Press the button and flash. So right now there’s, code on the GitHub that has the serial interface running in Python.
So a little more involved. The other thing too is because it’s the serial interface, you really need an isolated. Opto isolated interface because it’s being plugged into your [00:25:00] wall while it’s running. But yeah, it’s not the other thing too with that is that one of the things I realized is that you look at any of the people doing these repairs, everyone just replaces the boards, right?
There’s no firmware update process at all. So there’s this, there’s a connector on the board for doing the firmware update. It’s not documented that I’ve ever been able to find on the service side, even which is also pretty crazy because it’s to either get a newer firmware update.
And… There, I don’t know what the difference is because I ended up erasing my oven, but I could see the checksum of my original oven firmware is different from the checksum of the new oven firmware.
And it’s the
Paul Roberts: been a, there’s been a, modification at some point,
Colin O’Flynn: yeah, right,
Paul Roberts: right,
Colin O’Flynn: fixed some of the issues, like it didn’t seem everything, but
Paul Roberts: right,
Colin O’Flynn: sense, right? That they would have had newer versions. But so the thing is I’m sure people are replacing their boards. They don’t even need a pack or anything, right? They just want a firmware update. And, yeah I’m [00:26:00] curious because the micro controls EOL, it also has the question, are they going to stop making more of the boards?
Paul Roberts: Right?
Colin O’Flynn: Are people not going to be able to get spare parts for their oven?
Paul Roberts: right.
Colin O’Flynn: maybe all it needs is a firmware update.
Paul Roberts: Firmware update. You don’t even need a new board, right? You can just do this. Yeah, Totally without. And, as obviously replacing the board, even though it’s better than replacing the entire oven, but you’re still creating a fair amount of e waste when you could solve this problem much more simply with a software update.
Colin O’Flynn: and it’s and, the software maybe that’s something people can do if they’re techie and have a serial interface thing, it’s they…
Paul Roberts: Or here’s a crazy idea. Maybe oven makers can start to make it easy to do this type of stuff, right? Because that will prolong the life of the device. So that brings us to the sort of the other aspect of your talk that I think is so interesting. And I’m so glad Black Hat took you is, presenting you, which is this whole.
Larger conversation around repairability, [00:27:00] circularity prolonging the useful life of devices and you point out in their presentation that, one of the consequences of this funky temperature sensor feature is that a lot of consumers who aren’t PhDs in electrical engineering are going to say, I’m not going to reverse engineer this and patch the code. I’m going to throw the I’m going to get rid of this thing, throw it in the trash, throw it in a landfill, and get a new oven that doesn’t have this problem. Whereas, as you’ve shown, you don’t need to do that, right? There’s a way to get to correct the workings of this software and get the equipment to be much more functional than it is, as shipped by the manufacturer.
So I’m really is there a business model, right? For what you did. You talked about if we had a dongle and so on is that a possibility? If not with this device [00:28:00] maybe, other devices are going forward.
Colin O’Flynn: I’m sure there must be I’m terrible with this cause I always have ideas that are 60% executed. So I of course I thought of a little bit about this and I was like, Oh no, Now something else is interesting. So no time for that. I’m sure there would be I know, for example, on the automotive side, there’s tools for reflashing like airbag computers and things because once the airbags go off, the computer becomes invalid. And so I know there’s people that sell tools and that’s their whole purpose is it’s physically fine. You just need to reflash it. And they charge people to every time, I don’t know how exactly does a flash or there’s a big cost of getting the tool. I’m sure for oven repair techs, like again, because Samsung doesn’t even tell you that this is possible to reflash them.
Or if you wanted the, code added to debug so that you could actually see, Hey, what is the temperature sensor? You don’t have to go replace everything. You just plug in
Paul Roberts: right.
Colin O’Flynn: can dump, [00:29:00] Oh, the temperature’s off. There you go. I know to replace that, not spend all this time.
And to your point of people replacing them, I’m sure there’s lots of techs that have come over when people have this oven, they say they have this problem. And this tech’s going to say I can try to fix it, but I’m going to have to replace the temperature sensor. And if it’s not that, 50 bucks.
I’m going to replace the control board. That’s 200 and it could be the element. Oh, there’s a hundred. You’re going to be 500 in and it might not fix it. I can’t even guarantees
I’m positive they’ve had that conversation because the tech doesn’t want to most of the time they don’t like just throwing parts at it and…
Paul Roberts: and they’ll do what they’re trained to do, right? Most techs are going to run down a checklist of likely problems and what’s most likely and start there. If they’re not trained to say, oh, hey there’s this logical port you can flash the software.
Here’s an update, then they’re not going to do that. You don’t do what [00:30:00] you’re not trained to do.
Colin O’Flynn: Yeah. I mean, I’m exactly right. That’s it. You’re not going to say, “Oh, let’s get this random guy’s code.” And “don’t worry customer, I trust him.”
Paul Roberts: Talk about, so how does it work now? How did it work before with the temperature sensing and how does it work post Colin patch?
Colin O’Flynn: Yeah, so the, main issues, the number one issue with this seems to be, they basically I mentioned at the beginning, I thought maybe they’d do this PID controller, which would be your standard, right? Really nice temperature controller system. What they do instead, and my friend that’s more into industrial told me this once, and then I, now I’ve forgotten what he told me exactly, of course.
People are going to listen, say that’s not quite right, but it’s a pulse control, right? So it’s like it has a fixed pulse with heater and you can see it just does these pulses
And I’m guessing I don’t know if that’s because the element you don’t want it on all the time or whatever reason But the issue with that is that the pulses are too narrow to actually [00:31:00] recover So if you put a big load in a big turkey and the doors open for a few minutes, so the temperatures dropped it can never, it just, it doesn’t have enough power in that mode to
Paul Roberts: right,
Colin O’Flynn: In preheat mode, it just turns on the element more or less solid. So in that mode, it gets up to temperature really fast and then it tries to just maintain and it’s the maintenance mode that’s problematic on mine and it could be. So the other thing you can get is if the element is a bit broken or my theory is it’s a bit worn over these are older ovens, it’s aged.
The kind of parameters they tuned for have shifted.
Paul Roberts: No longer.
Colin O’Flynn: if this was a PID controller or something that had more feedback in it, I think it would recover more reasonably. Those are designed to account for this. So that was the fundamental problem was it just wouldn’t get back up to temperature.
If you had a big load in and it seemed to be because of this, and what I did is I [00:32:00] basically leave my oven in preheat mode now or the firmware, so it’s doing much more aggressive pulses. So the time it turns on for his longer it works. It looks a lot more like a classic mechanical thermostat.
It’s over temp do nothing. It’s under temp turn on. So you get a bit more of a ripple, but it’s averaging on what you want instead of before where it was slowly going down.
Paul Roberts: And here’s the big question. You’ve done a couple of tests. I noticed shepherd’s pie, some souffle. What’s the what’s the experience been?
Colin O’Flynn: it seems good. To be honest, we had to recalibrate a bit because we without realizing that you get so used to the fact that it’s we’ve been cooking stuff hotter.
Paul Roberts: Hotter. right? Yeah.
Colin O’Flynn: to recalibrate a bit with that. Yeah. And, the other issue right now is still this problem.
And it only seems to happen when you first turn it on where it just won’t heat it all. Needs to be rebooted. So [00:33:00]
Paul Roberts: You need to,
Colin O’Flynn: and a part of it might be, I did add this monitor code I mentioned, which was more serious it uses some memory that might be used elsewhere. I took a guess.
Paul Roberts: Right.
Colin O’Flynn: So I may just remove that obviously for cooking. I don’t need the monitor. That was purely for debug and testing.
Paul Roberts: One of the things that’s really interesting to me about this research is I think it, it speaks to how the sausage is being made right now in the the appliance world, the personal electronics world, which is, I think we all have this, Sense of manufacturers like Samsung or LG or even Apple “Oh, that mean they’re always using the best components and they’re, they’ve got the smartest software engineers working for them, which like at a high level is true, but it’s like Upton Sinclair’s The Jungle, right? When you go into the sausage factory onto the floor and see how it’s being made, like in this case, you’re like they’re using [00:34:00] a really old microcontroller with a really old processor that can’t really handle that much. And they’re actually having to make design choices about this software based on the limitations of the hardware.
And that’s. They’re making compromises, right? And like you said Oh they’ve, got this port. They could be doing software updates and that type of maintenance, but they’re not they’re not doing it. They’re not training or techs to do it. So they’ve, structured this whole kind of product lifestyle cycle in a way that’s very suboptimal, right?
would look at that and be like you’re really, you’re not really doing everything you could be doing to support this thing and, in fact, you made a whole bunch of junky design decisions based on this junky old hardware it’s like the sausage factory you’re like, Hey you just pick that piece of meat up off the floor and put it in the sausage…. You can’t do that And your research really, for me synthesizes all that yeah [00:35:00] like, when people like you peel back the covers of the software locks and the DRM and actually look at what’s going on, often it is this kind of Upton Sinclair’s The Jungle, like, oof what a mess.
Colin O’Flynn: yeah, no it’s, interesting because so coming maybe more from the design engineering side too, right? There’s a lot of these things where especially in the embedded space. It’s funny people that don’t have the design embedded design background get into it. And you’ll see their talks and they’ll be like, Oh, like the embedded designers, why did they do this crazy thing?
That’s totally terrible and insecure. It’s
Oh, I know because I would do that exact same thing because I just took the manufacturer’s example app and compiled it and was done.
Paul Roberts: Made a
Colin O’Flynn: Right. Like I’ve I’ve, definitely done that. No question. But yeah, and and sometimes it’s there’s constraints within the company. So a while back I did some stuff looking at Phillips Hue and they had this issue where their bulbs were using a shared
key. [00:36:00] Exactly. Yeah. And they use the same key across all the bulbs, right? For the, firmware which, isn’t a great idea. And the thing is, they go to the server and they ask Hey, server, give me the latest version of firmware for this product.
So they 100% could use a per device key. And they have it. Good. It’s like smart engineers working there. You know them and, I never actually asked in the end, but I 100% know, right? Like a bunch of engineers. I’m sure we’re like, Oh, this is a bad idea. We should be doing per device key. Of course, or at least per product line key or something.
And I’m guessing it was just a cost, right? Cost of maintaining. This larger update infrastructure securely doing or more securely doing it. It’s also a lot of the case where people are giving these designers impossible constraints, right? Either end. The cost for the consumer product has to be below X, right?
Even if it was 10 cents more and they could Add this more secure device or whatever it might [00:37:00] be. Yeah, it’s interesting too, because clearly, and you can never know for sure, of course, not being in the company, but a lot of the time it looks like this was an artificial constraint that the engineers made work.
poor choice.
Paul Roberts: the process of making these products and writing the software, like you got a lot of things that you need to do, and if there isn’t a really top down process of saying let’s go through what you’ve done with a security eye, with a sort of red teaming eye on how would a malicious actor look at this?
What types of things would they be probing around at then? It’s easy for stuff to slip off your radar as a engineer, right? You’re trying to just get features done, check them off, and move on.
Colin O’Flynn: Yeah, completely.
Paul Roberts: Yeah. You, have you had any contact with the manufacturer, with Samsung or Samsung engineers or anything?
No.
Colin O’Flynn: yet. I, didn’t I a while back I had reached out [00:38:00] but I, didn’t push that very hard. It’s such a a, the microcontroller on the Toshiba side is EOL for,
Paul Roberts: Yeah.
Colin O’Flynn: a while. So in fairness. You They shouldn’t.
Paul Roberts: So let me ask you what, is the big, what is the big fix for, this in, in, in your mind? I, again the high, the 10, 000 foot level version of your, of what you’ve Explored here is the manufacturer put not very high quality software on their home appliance. It was causing problems for owners, but there was no really easy way to fix it.
You just happen to be technical enough to actually go in and fix it yourself. What is, what in your mind would be like the big fix for this across product categories, across the economy these types of Problems that when you really look under the covers, they’re really software problems and the manufacturer is often the source of the problem, not the solution, or maybe [00:39:00] the source and the solution.
Colin O’Flynn: yeah, it’s a good question. If you could solve all of that, you’re really solving poor software design in general. For these, I think for me, the,
Paul Roberts: Good luck with that.
Colin O’Flynn: yeah and and, I guess maybe it speaks to lots of the question of how even know if a product does what it says it should do, right?
Consumer protection right is basically it, which is maybe easier when it’s a physical, I don’t know, a door that doesn’t open or it catches and in a fire your door doesn’t open. There’s gonna be a recall, but I think software has become so and nebulous that Yeah what is the way that consumers can say this class action they were trying to do is really how, I guess people are trying to solve it that way because Like you look at the complaints have been going on for years.
and
Paul Roberts: The other is just to have. Independent [00:40:00] software providers, right? Like, you who, who can come along and say, Hey Samsung, sold do an oven with crappy software on it. We’ve got an update we’ve got a version of software that will not only fix that problem, but add this feature and that feature and maximize the potential of your hardware, right?
But we don’t really have that so much.
Colin O’Flynn: Yeah,
Paul Roberts: piece of hardware, and you just assume that it’s always going to run the software that it came with.
Colin O’Flynn: Yeah, no, you’re completely right. And I originally I had thought of Ooh, could I make an open source oven controller for my oven software and then have a better pay
Paul Roberts: Do it Do it!
Colin O’Flynn: So I’ll see the issue is that there’s then I realized when I realized ah, then there’s the variants of this other microcontroller.
Although if you had an abstraction layer, it might not actually be too bad because it’s to be honest. Most of the buttons on my oven, at least it has all these buttons. I’ve never pressed like chicken tenders [00:41:00] button and all this stuff. Most of the time, the only feature I use is
Paul Roberts: Bake. Yeah. Yeah.
Bake and broil.
maybe. Right. Yeah.
Colin O’Flynn: what you’d have to implement.
I don’t think that much,
Paul Roberts: Final question is, if other people are listening to this and Oh, this is really cool. I want to get more into learning about the embedded software that might be running on my oven or my dishwasher. Like, where would you suggest that they go and start digging around?
Colin O’Flynn: Yes, I mean if they’re interested, so I’ve posted a bunch of stuff specifically to my oven which might be really interesting to people just to see, hey, what’s this actually look like? So my blog, colinoflynn.com, has a link to that, to the GitHub. There’s a bunch, you a bunch of links below it.
So it’ll be at Black Hat. I also co authored a book, the Hardware Hacking Handbook. So Jasper, who’s at a company, Riskier, that does very similar stuff at a higher level. And I wrote this book and it talks a lot about how do you, even get into it in general? Which is like [00:42:00] ripping the cover off and, playing around. And, for a lot of this stuff, it is just as a kid taking things apart and looking at what it, what’s inside it. With the caveat, eBay is really nice for this because as in my example, the oven control board and oven is really expensive.
So you can buy an eBay one, poke at that before you, yeah, get your partner less happy with you for
Paul Roberts: Colin O’Flynn thank you so much for coming on and speaking to us on the Security Ledger podcast.
Colin O’Flynn: Thanks so much for having me, Paul. It’s been a lot of fun.
