Our Top 3 DevOps Challenges (& Why Disruptive Technology Is Required)

It’s been a few years since Cloudify strengthened its focus on DevOps and IT, and throughout this time, it has become pretty clear to me that the trends we’re seeing across our client base represent a bigger, industry-wide problem. We’re at a time of accelerated transformation to the cloud (COVID-19 certainly provided a much needed boost – but we’ll leave this for another blog post), and I strongly believe that the insights below should assist DevOps and IT leaders in making informed decisions related to their cloud transformation strategy. The following trends, in my opinion, should be top of mind when planning ahead:

1) Hiring DevOps Talent Is Harder Than You Think

If your company is actively looking for DevOps talent, what I’m about to say next won’t come as a surprise. Hiring DevOps these days is extremely challenging. There’s an estimated shortage of ~40M Software and DevOps engineers globally, which means DevOps leaders would most likely not be able to meet their team’s growth plans and in turn, teams will not be able to address the scaling requirements of the business. I strongly believe that software is the only valid workaround for this issue – more specifically, automation software. Your DevOps org needs to templatize and automate as many environment creations as possible and gradually replace manual work with software, there’s no other way; which brings us to our second challenge…

2) DevOps Automation Is Unfortunately Broken

Terraform, Ansible, Kubernetes, AWS CloudFormation, Microsoft Azure Arm, Red Hat OpenShift, GitOps, Jenkins, and the list goes on. Back in 2020, Catalyst Investors shared a presentation called “Evolution of the Software Development Lifecycle and DevOps Tools”, and slide #4 took me by surprise.  Apple is using 74 different DevOps tools. Stripe and Google? 59.  Docker? They’re using 55 different DevOps tools. How does one integrate and turn this spaghetti of tools into an end-to-end automated environment? The answer is ‘one does not’. We are seeing across our customers that the place where everything supposedly comes together is the CI/CD system, usually by scripts or manual work. We recently asked a select number of clients how long it takes them to spin up a typical environment. The answers were between 2-3 weeks for a small environment (a single VM running a simple app), and 4-6 weeks for a complex environment (a highly-available web environment on AWS). In other words – if it takes over a month (!) to spin up one development environment – scaling your cloud operations will obviously be an issue from the get-go.

3) The Disconnect Between DevOps And IT Is Real (And Dangerous)

Most of our medium and large customers are describing the following challenge: Their business is using an ITSM (IT Service Management) system (ServiceNow in most cases) to gather requests from across the org-  including R&D requests for development environments. This in turn requires constant integrations into the ITSM, which significantly slows down the process (see environment spin-up time above). This disconnect continues – while the DevOps team uses mostly CI/CD to publish software and infrastructure updates, IT systems are not always up to date, and therefore can expose a very real risk of overriding a change made by DevOps teams as a result. This brings me back to the need for software, as opposed to human intervention as discussed on challenge #1. There needs to be a state-aware system up at the top which integrates to both IT (ITSM) and DevOps (CI/CD) systems on one hand, and as many automation tools as possible on the other – (see this blog post as a reference). The end result? We call it an ‘Environment as a Service’.