Software’s Sausage Factory: The Supply Chain

Sourcemap Laptop Map
Far flung supply chains for hardware and software can pose security risks for tech firms and their customers. (Image courtesy of sourcemap.com)

In-brief: Experts warn that supply chain insecurity run broad and deep, threatening the security and integrity of technology dependent organizations.

Last week, Christian Science Monitor ran a story I wrote on flaws in a common piece of network connected hardware: so-called Remote Power Management (RPM) devices. As my story noted: these are easily missed devices that are, essentially, Internet connected power outlets. They’re also riddled with exploitable security flaws, including hidden features that can be run without a user name and password. My story notes research by two firms: BorderHawk and Senrio (formerly Xipiter)* that uncovered similar flaws and, in one case, evidence that RPM devices deployed in a large energy firm had been compromised by unknown attackers. For my story, I also spoke with IT administrators at a large university and a municipality that had deployed these vulnerable devices in ways that made them vulnerable to remote attacks.


LEARN TO SECURE THE INDUSTRIAL INTERNET OF THINGS 

Trusted Computing Group has how-to and demos with Microsoft, GE, Infineon, OnBoard Security, Wibu-Systems at IoT Solutions World Congress. Get your free expo pass code 111B9B47 or discount conference pass code 526E24AF


Remote hackers being able to randomly shut the power off to critical equipment or IT systems? That’s scary stuff. But the bigger story may be how common security problems like those discovered in remote power management devices are. In fact, security experts that I spoke with said they are just one example of a vast and looming problem: a global technology supply chain that is nimble, but too-often churns out poorly designed and insecure products. That technology, in turn, is consumed by organizations around the world.

My story highlights problems discovered by Senrio researchers in remote power management devices by Arizona-based SynAccess. But the truth is that similar problems are common elsewhere and are rooted in the modern, global technology supply chain. The experts I spoke with agreed that embedded systems of all types: from networking equipment to industrial control systems and even medical devices may harbor hidden administrator accounts, undocumented features or outdated software with exploitable vulnerabilities.

Markets for components in places like Shenzhen, China, are one-stop shopping for tech firms. But quality is an issue.
Markets for components in places like Shenzhen, China, are one-stop shopping for tech firms. But quality is an issue.

“We see lots of different devices, but a lot of the same problems,” said Billy Rios, the CEO of the security start up Whitescope. Rios, a noted security researcher, says that undocumented features like those found in the remote power management devices can be found in the software that runs many embedded systems – from medical equipment to assembly lines.

Weak authentication is the norm and the use of varying and inconsistent internal components is surprisingly common. Also common are hidden, “backdoor” accounts created for use by technicians but, of course, open to anyone who knows about them, Rios said.

Far Flung Partners…and Problems

The problem is a byproduct of changes in the way that technology firms source and build their products. Computer manufacturers like Dell and HP long ago shifted from domestic suppliers and distributors to lower cost alternatives in countries like China.

Matt Caldwell of the firm BorderHawk notes that computer products 25 years ago were assembled in Texas from parts made in Silicon Valley and shipped directly to retail stores and companies in the U.S. Today, finished products are made of parts manufactured in China, Taiwan, the Philippines and Indonesia, assembled in China and shipped via a web of importers and distributors to stores and customers. That shift in production also engendered a huge shift in trust, Caldwell told an audience at the S4 Conference in Miami in January.

But the change is more profound than just a shift in manufacturing to low wage countries, observes Joe Grand of Grand Idea Studio, a noted electrical engineer, inventor, and hardware hacker. Hardware manufacturing, unlike software design, has become a commodity service, with the result that technology companies pay much less attention to the hardware that will run their software than to the software itself.

There are exceptions, of course. Image- and quality conscious firms like Apple still tend to both the software and hardware that they sell, carefully sourcing each, individual component. But Grand and others say that it is increasingly common for western technology firms to expend a disproportionate amount of time and energy on the software side of the product equation, with little thought to the platform that will run it.

The Trouble with Turn Key

Hardware vendors today try to make their products ‘turn key,’ bundling components and including sample software (or firmware) to run it, all in an effort to shorten the path to the customer’s door. “They just buy ‘the hardware’ from a vendor that meets their specifications and that’s just accepted as ‘good,’” said Grand. “Whatever hardware is in it, whatever software it’s running, that just goes into the final product.”

Packages might include the hardware chassis, the layout and schematics of internal components, an operating system with pre-programmed functions and sample code upon which new functions can be built can be purchased as a piece. Engineers can then customize a bit based on that, adding or removing functionality, and be ready to ship Grand said. “Vendors want to make it easy for engineers to use their product,” he said. “So someone buy off the shelf software and does some minor customizations of the sample code.”

That approach makes technology firms today far more nimble than their predecessors ten or 20 years ago. It reduces the cost of manufacturing and time to market. But it also increases the risks to the company and its customers. The problem comes when low quality – or compromised – hardware and software are part of the mix.Individual components used in a product may be of low quality, or may be infected with malware or other malicious code. Just as often, a shiny veneer on high tech equipment hides decidedly down market ingredients.

Supply Chain Problems Float Downstream

Such dangers can be spread widely through the supply chain and be very hard for customers to spot. For example, there were warnings in March about a serious privilege escalation vulnerability in firmware for a common ARM SOC (system on chip) made by the Chinese vendor Allwinner. The vulnerability impacted a wide range of ARM development boards, which, in turn, are the foundation for countless connected gadgets. “The main problem is that companies don’t have any thought about whether some kind of malicious event might happen,” said Grand.

Stephen Ridley at the firm Senrio said that when his researchers took the NetBooter RPM device apart, they found a relatively simple product running on a low-power PIC micro controller. Ridley said the NetBooter’s PIC chipset is a favorite of hardware hobbyists, but lacks the power and security features of more advanced chipsets like ARM. “It’s considered very simple and low cost. That’s probably why it was used,” he said.

“Hardware is a misunderstood, unknown territory.” – Joe Grand, Grand Idea Studios

Caldwell said his company’s analysis of the compromised RPM devices they discovered raised similar concerns. They were labeled “Made in the USA,” but were clearly sourced over seas. Misspellings graced the product label on the outside and certificates for compliance were no longer valid. Border Hawk bought multiple units of the same device, but found that each contained a slightly different mix of hardware inside – a problem not dissimilar to that faced by the meat packing industry in the early part of the 20th century. “Hardware is a misunderstood, unknown territory,” said Grand. “It’s still trusted a lot of the time. People buy a piece of hardware and take it for granted. They assume it is secure. They assume it does what it does and only does what it does.”

There is no easy fix for the problem. Grand advises companies that are developing new embedded systems to build relationships with suppliers. “The way the Internet is, you can order SOC (system on chip) devices or an embedded system running Linux and not ever meet the people who are making it,” he said.

Likening the process to the current craze for locally grown food, Grand says that technology firms need to know their suppliers personally. “If I’m sourcing a module, I want to go and see where its made. I want to make sure it’s a legitimate package and that the company meets my standards.” Global supply chain or no, it’s human relationships that matter, Grand said.”That’s what’s missing in this generation of products,” he said. “We’re missing an important part when we take the human out of loop.”

(*) Clarification: Links and references to Xipiter have been changed to Senrio (senr.io), to reflect the company’s recent name change. PFR 5/27/2016.

Comments are closed.