This is the first of a new five-part series of articles developed by TAG Cyber in conjunction with Sicura to offer insights and guidance into modern DevOps security using automated and integrated support.
Cybersecurity threats to DevOps emerge during the pre-delivery phases of code development (the Dev portion of the pipeline) as well as during the post-delivery execution phases (the Ops portion of the pipeline). Both aspects of DevOps security demand modern solutions that reduce risk without degrading the primary goals of agile software development. That is, the security cannot slow down development or impede deployment and execution.
In this article, we cover the key aspects of both types of DevOps security threats. For the pre-delivery phases, we explain the type of modern challenges that emerge for development, test, and staging teams, including the possibility of compromised insiders. For the post-delivery phases, we cover the security issues that emerge during deployment, operation, maintenance, and support.
Figure 1-1. Threats to Pre-Delivery and Post-Delivery
The DevOps model fundamentally changes the way the development, operations, and business teams work together. The legacy multi-month development cycles have been replaced with short, easily achievable developer deliverable timeframes, usually spanning a couple of weeks. These so-called sprints leave little time for the inclusion of legacy security testing practices, resulting in security teams being excluded from the process.
While there has always been cultural pushback from development teams regarding inclusion of security in the development process, the DevOps model actually increases the complexity of testing significantly. In the new model, only parts of the application code are updated to add new functionality, and there could realistically be updates daily depending on the number of teams working on it. The old model of ‘application scanning before production’ doesn't fit within a DevOps model, and many security teams are struggling with how to shoehorn their toolsets into the new approach.
Another significant change in the pre-delivery aspect of a DevOps model is the ability of the development team to stand up complete infrastructures, which historically relied on other IT infrastructure teams. The concept of ‘Serverless Computing’ does not mean servers do not exist - it means only they are managed in the cloud by the providers. Since the development teams can now instantiate a server, a database, and the appropriate supporting applications with zero knowledge of the security teams highlights the significant pre-delivery risks that can be introduced into any infrastructure.
Additionally, since infrastructure relies on user accounts (human and machine) and cross-server access, the risk of unmanaged, high-privilege accounts being created and shared is high. This risk is elevated even further when you consider the sharing of user accounts under the guise of ‘collaboration’. Account credentials, SSH keys, and MFA tokens will all be shared among the DevOps teams to ensure the sprint deadlines are achieved. Again, since all device and user access are controlled by the developers standing up the infrastructure, speed and ease of access will always take priority over secure access and user controls.
Unlike most enterprise infrastructures, cloud and DevOps deployments are not designed to be centrally managed. A developer with a credit card can easily stand up an entire environment for a new customer application within a few days, not the weeks (or months) it used to take. While the efficiency and speed to market is valued by the business, the potentially catastrophic breach of personal information is often disregarded until it is too late.
If we consider a modern cloud deployment, for example, there is no manageable, dedicated DNS infrastructure that needs to be updated or a firewall whose ports need to be opened by the networking team. Long gone are the legacy security controls that protected the entire infrastructure, so even when unapproved devices did make it onto the network, they had some semblance of security controls protecting them.
With today's cloud model, there is no ‘IP Assignment Process’ or ‘Firewall Change Request’, that were typically the catch-all for rouge developers and devices. In the cloud world, one can have a fully functional LAMP (Linux-Apache-MySQL-PHP) stack on the internet within minutes. If a developer wants to open insecure protocols or call APIs from untrusted third parties, there is nothing stopping them.
This truly comes into play when we look at many of the recent major issues that as an industry, we have been dealing with. Without an accurate asset inventory, it's impossible to know if you're exposed to threats such as Log4j or Log2Shell. Patching and configuration management are highly visual victims of the DevOps movement.
Another oft-overlooked side effect risk of DevOps is that, since the developers own all of the infrastructure, they are, by default, the support team as well. So, when an outage is reported, the developer will fix it from wherever they are. If the enterprise is lucky and the developer had set up an SSH connection, rather than just Telnet, they would simply connect and fix the issue. While this is fantastic for the business, those SSH keys are likely never changed or revoked, leaving that developer with full access for as long as the keys are valid.
One issue more catastrophic than inventory gaps involves the risk posed by unmanaged users and accounts within a DevOps environment. Recalling that speed and flexibility are key to a successful DevOps sprint, the last thing developers will allow to get in their way are passwords. Common shared passwords across users and systems are a frequent deployment method to ensure everyone who needs access can get it. While in a development environment this is slightly more acceptable, DevOps is a write-build-deploy-test cycle, which inherently means all of those shared, weak passwords and unmanaged accounts will likely end up in production.
In many ways, the DevOps model runs parallel to what many security teams faced a decade ago. From a complete lack of device inventory to misconfigured and unsupported devices with unauthorized and unmanaged users, the cyber risks surrounding a haphazard DevOps implementation are a real concern for most enterprises.
Again, if we look at all of these risks layered upon one another, the potential for a significant breach becomes apparent. A developer with full access to production customer data could make a significant amount of money selling that data on the underground markets. The lack of patching, configuration management, and user access are fundamental risk mitigation practices that could easily introduce significant risk within an enterprise, all without the security team’s knowledge.
While the potential risk of unmanaged DevOps is quite high, there are many approaches and tools available to both developers and security teams to mitigate most of it, all while supporting the goals of the DevOps sprint. In the second article, we will look at how we can mitigate these risks using automation and thus ensure that the security team is a welcomed addition to the DevOps program.
by John Masserini, Senior Analyst, TAG Cyber
About TAG Cyber
TAG Cyber is a trusted cyber security research analyst firm, providing unbiased industry insights and recommendations to security solution providers and Fortune 100 enterprises. Founded in 2016 by Dr. Edward Amoroso, former SVP/CSO of AT&T, the company bucks the trend of pay-for-play research by offering in-depth research, market analysis, consulting, and personalized content based on hundreds of engagements with clients and non-clients alike—all from a former practitioner perspective.
Copyright © 2022 TAG Cyber LLC. This report may not be reproduced, distributed, or shared without TAG Cyber’s written permission. The material in this report is comprised of the opinions of the TAG Cyber analysts and is not to be interpreted as consisting of factual assertions. All warranties regarding the correctness, usefulness, accuracy, or completeness of this report are disclaimed herein.
Next week we'll be back with Part 2 of the TAG Cyber Sicura series, on the role of automation in DevOps security. Interested in learning how can Sicura can make your environment more secure and your DevOps team more efficient? Get in touch.
Ready to maintain secure and compliant infrastructure focused on cost, automation, and consistency?