The deployment of DevOps tools and platforms at many organizations recalls the bad old days of Windows XP, with lax control of authentication, loose configuration and scant attention to security, experts warn.
The Jenkins continuous integration (CI) tool is the heart and soul of many DevOps environments, but it may also be a huge, lurking security risk for development organizations, the head of research at the firm CyberArk* warns.
Jenkins deployments drive cutting edge agile development operations, but too often these shops employ security measures that are inadequate and reminiscent of the ‘bad old days’ of Windows XP, said Lavi Lazarovits, the Group Research Manager at CyberArk Labs, part of the security firm CyberArk.
“Windows XP was infamous for its lack of security, but now we’re experiencing the same thing with emerging technologies in cloud and DevOps,” Lazarovits told Security Ledger on the sidelines of the CyberArk Impact event in Chicago last week.
New platforms, old problems
As was the case with Windows XP, the security problems with platforms like Jenkins are often the product of misconfigurations of critical features and lax user- and account permissions, rather than exploitable security vulnerabilities in the underlying platform.
CyberArk researchers frequently find that security is not enabled by default on Jenkins and other platforms. “Breaches often happen because of simple misconfigurations. They’re not something that is extremely difficult to exploit,” Lazarovits said. For example: Jenkins deployments can often be identified by public Internet search using platforms like Shodan. Jenkins deployments may be configured to allow any authenticated user to have broad permissions on the platform rather than using granular access policies based on a user’s role and responsibilities. Some even allow anonymous users to authenticate to the platform – an open invitation to hackers, Lazarovits said.
Other security issues are more subtle. In a presentation at the CyberArk Impact event in Chicago last week, Lazarovits was one of a number of researchers warning that risks lurk in the complexity of many DevOps and cloud platforms. For example, organizations often fail to properly isolate both Jenkins master servers and distributed “worker” machines or monitor them for unusual activity, he said. “Any privileged access to both master and workers is very important,” he said.
Continuous integration and deployment (CICD) platforms like Jenkins greatly accelerate the work of development teams by automating a wide range of once-manual developer tasks. But “automation relies on trust,” Lazarovits notes. “Jenkins trusts its workers. The Git server trusts Jenkins, and so on.” All that trust can become a slippery slope to insecure deployments and exposed data.
Adding to the problem is the fact that it often falls to developers and operational teams, not security teams, to deploy and manage DevOps platforms like Jenkins. Operational groups often lack security expertise and focus on availability and productivity, rather than system integrity. Presented with a choice between productivity and the implementation of a security feature, the security is often what gets sacrificed – especially where automation is being used, Lazarovits said.
Jenkins isn’t the only platform to pose security challenges to DevOps organizations. Asaf Hecht, a team leader in CyberArk’s security research group, addressed the risk that so-called “shadow administrators” pose on platforms like Amazon AWS. There, roles that appear to have limited rights may actually have extensive permissions beyond what their name would suggest, Hecht warned. For example: a seemingly limited administrator role charged with assigning user roles is actually a super-administrator capable of elevating or revoking administrative privileges to any other user.
Similarly, the Kubernetes container platform has an infinitely flexible authorization scheme that makes it easy to lose track of who has access to what. “It is so complicated, the chances that administrators know where their privileged tokens actually are is nearly zero,” Lazarovits said.
Security before velocity
Lazarovits’s advice to organizations that use Jenkins heavily is to put security before the velocity of development. “Have your security team work with security tools to secure the environment,” he said. “Don’t allow development engineers to do it.”
Security teams should also work with development and operational teams to gain greater visibility into the operation of DevOps platforms like Jenkins. “Jenkins has great tools that provide visibility, but most DevOps organizations are far behind in monitoring the activity on platforms like Jenkins, Lazarovits said. That’s especially true when compared with the scrutiny they apply to ordinary network activity.
If nothing else, development organizations need to establish clear visibility into who has privileged access within their DevOps environment, what credentials are being used and who is using them.
“These are simple questions to ask, but not so simple to answer,” he said. “Organizations just need to have a clear set of priorities, so they know where to start.”
(*) Disclosure: Security Ledger’s coverage of Impact 2019 was sponsored by CyberArk. For more information on how Security Ledger works with its sponsors and sponsored content on Security Ledger, check out our About Security Ledger page on sponsorships and sponsor relations.