SolarWinds Hacked Image

What SolarWinds Tells Us About Securing the Software Development Supply Chain

The recent SolarWinds attack highlights an Achilles heel for enterprises: software updates for critical enterprise applications. Digital signing of code is one solution, but organizations need to modernize their code signing processes to prioritize security and integrity and align with DevOps best practices, writes Brian Trzupek the Senior Vice President of Products at DigiCert in this thought leadership article.


Even in today’s security-charged world, the SolarWinds breach was a wakeup call for cybersecurity professionals. It was distinguished by its sophistication and the fact that it was carried out as part of legitimate software updates. The incident was quickly all over the news and has brought renewed focus on need for secure DevOps.

Brian Trzupek of DigiCert
Brian Trzupek is Senior Vice President of Products at DigiCert

The SolarWinds attack took advantage of what is usually a best practice: regular software updates. It was based on malware inserted into the development process for network monitoring software, which created a back door. When the software was updated, the software, including the back door, was distributed into customer environments, compromising a broad array of prominent U.S. government systems. Once in place, it enabled the hackers to quietly spy on them.

Also read: “Automating your way out of PKI chaos.”

Code Signing Alone Is Not Enough

The security incident at SolarWinds was especially unsettling because could easily happen at any large software organization that delivers regular updates. Code signing certificates are a highly effective way to ensure that software is not compromised. However, it is only as effective as the strategy and best practices behind it. When poorly implemented, code signing loses its effectiveness in mitigating risk for software publishers and users. Issues often include:

  • Using the same signing key to sign all files, across multiple product lines and businesses
  • Lack of mechanisms in place to control who can sign specific files
  • Insufficient reporting capabilities for insights into who signed what and when
  • Failure to sign code at every stage of development, as part of an overall security by design process
  • Lack of signing and verifying code from third parties
  • Poor processes for securing keys and updating them to new key size or algorithm requirements
  • Failure to test code integrity before signing
  • Inadequate visibility into where certificates are, and how they are managed

Common pitfalls might include using the same signing key to sign all files, across multiple product lines and businesses. Some organizations might have no mechanisms in place to control who can sign specific files. They may also lack reporting capabilities, which can provide insights into who signed what—and when.

[Read Brian’s piece Staying Secure Through 5G Migration.]

What have we learned from the SolarWinds attack? For organizations where DevOps is fundamental, applying best practices to signing processes is more essential than ever. According to some studies, more than half of IT security professionals are concerned about bad actors forging or stealing certificates to sign code—but fewer than a third enforce code signing policies on a consistent basis. It’s time for organizations to do better and enforce zero-trust
across all their systems, signing everything, at every stage after verifying it is secure.

Simplifying and standardizing

Traditional code signing processes can be complex and difficult to enforce. They are often based on storing keys on desktops as well as sharing them. Visibility into activities is often limited, making mismanagement or flawed processes difficult to discover and track. To mitigate these issues, many organizations are simplifying their processes using code-signing- as-a-service approaches. Code-signing-as-a-service can accelerate the steps required to get code signed, while making it easier to keep code secure. A robust solution can empower organizations with automation, enabling teams to minimize manual steps and accelerate signing processes. APIs can enable it to integrate seamlessly with development workflows and automated scheduling capabilities enable organizations to proactively and approve signature windows to support new releases and updates.

To strengthen accountability throughout the process, administrators can apply permission- based access. Strictly controlling access helps improve visibility into which users are allowed to sign code and which certificates and private keys they are allowed to utilize.

Standardizing workflows

Standardizing code signing workflows can also help reduce risk to an organization. Instead of allowing everyone in an organization to use the same key for signing, many organizations are using separate code signing keys for different DevOps teams, while granting administrators visibility over key usage. This best practice helps minimize the risk of mistakes that can occur across a company by limiting the ability of breaches to propagate. For example, if a key is used to sign a release that has been compromised, only one team’s code will be impacted.

Maximum flexibility to minimize risk

Key management flexibility is another helpful best practice, reducing risks by enabling administrators to specify shorter certificate lifetimes, rotate keys and control keypairs. For example, Microsoft recognizes publishers that rotate keys with higher reputation levels. With the right key management approach, an administrator could specify a specific number of days or months exclusively for files designed for use in Microsoft operating systems.

Secure key storage offline except during signing events

Taking keys offline is another measure that can secure code signing. With the right code signing administrative tool, administrators can place keys in a “offline mode,” making it impossible to use them to sign releases without the proper level of permission in advance. Release planning is a fundamental to software development, so most developers are comfortable scheduling signatures for specific keys directly into their processes.

Taking keys offline is a strong step to ensure that keys will not be used in situations where they should not be. It also adds a layer of organizational security by splitting responsibilities between signers and those who approve them—while providing improved visibility into which keys are signed by whom.

Freeing up developers to do what they do best

It’s clear that safeguarding DevOps environments correctly remains challenging, but fortunately the right management tools can minimize hassles—and maximize protection. As we’ve discussed, automation is essential for applying security across CI/CD pipelines. Seek out a solution that can fit smoothly within workflows and free up engineers from individual steps required for cryptographic asset protection. The tool should make signing keys easily accessible when pushing code and automate signing of packages, binaries and containers on every merge to master when authorized. Organizations also need a process for testing code integrity before they sign. A centralized, effective signing management tool can handle the signing tasks, while integrating with other systems that perform necessary integrity tests. For key security, the solution should provide the option of storing the keys offline in virtual HSMs. During a signing event, it should enable developers to access the keys to sign with one click, then return them back to secure offline storage

DevOps pros work within a variety of environments, so the signing solution should support portable, flexible deployment models via SaaS or on a public or private data center. Businesses in every industry are becoming increasingly software-driven and the challenges to DevOps organizations won’t disappear anytime soon. However, with the right approach to code signing, organizations can dramatically strengthen their security posture, minimize their chances of becoming the next victim and ensure customer confidence in their solutions.


(*) Disclosure: This post was sponsored by Digicert. For more information on how Security Ledger works with its sponsors and sponsored content on Security Ledger, check out our About Security Ledger page on sponsorships and sponsor relations.