Solving the ‘Speed Paradox’ with Backstage and Cloudify: a Full-blown PaaS

The number of tools in our DevOps tool chain have grown tremendously over the past few years. The move to public cloud has only accelerated that trend, and we are now facing an explosion of new tools, each targeted to solve a specific need in our pipeline or stack.

Additionally, the number of cloud resources and options to manage those resources has grown even more. For example, on EC2 alone, AWS currently offers nearly 400 different types of instances with choices across storage options, networking and operating systems. Complicating this further is that users can choose from machines located in 24 regions and 77 availability zones around the world.

This gives rise to the “speed paradox”—where the faster you grow, the more fragmented and complex your software ecosystem becomes, which in turn slows everything down again.

Most DevOps teams, including DevOps “Rock Stars” such as Spotify, Airbnb, and Twilio development teams can’t keep up with the demand for more environments and are now finding themselves becoming the new bottleneck.

The Rise of the Internal Developer Platform (IDP)

One potential solution to the speed paradox is the Internal Developer Platform (IDP). As its name suggests, an IDP provides a single place where developers can find all the resources needed to run their development environment. An IDP allows developers to speed up their development processes by offering improvements in:

• Efficiency – simplifying the way developers get access to their development and testing infrastructure through a self-service experience;
• Consistency – providing a consistent way in which developers consume infrastructure resources across teams; and
• Visibility – providing a single place where developers can see all the development pipelines, workflows, and states that are associated with their specific environments.

In the words of Gartner, “Product teams often struggle due to disparate tools and disjointed workflows as they accelerate digital transformation. Software engineering leaders leading platform teams must establish internal, self-service developer portals to enable consistency and scale cloud, agile and DevOps initiatives.”

Lessons from the PaaS

The idea of having a developer platform to reduce infrastructure complexity isn’t new and is almost as old as the cloud itself. Previous-generation PaaS (think Google App Engine, Heroku Cloud Foundry) provided a fairly simple interface to write web applications by abstracting cloud resources for developers. This experiment failed for several reasons.

Perhaps the most obvious reason for this failure is covered in the following user experience that I highlighted back in 2013 in, “DevOps, PaaS And Everything In Between”:

“Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.”

You can summarize the reasons for the failure of previous-generation PaaS thusly:

• Too opinionated – it is difficult or impossible to control the way the platforms run their infrastructure resources. You had to be dependent on the platform provider for almost any customization.
• Limited to simple web applications and language choice – the previous generation of PaaS was targeting simple web applications while in reality modern SaaS applications are much more complex, and include many complex processes and backend services that don’t fit into that architecture.
• Maturity – the previous generation PaaS started when the cloud was still a moving target. The attempt to abstract such a moving target was doomed to fail.

What’s Changed?

At the end of the day, we’re trading simplicity for flexibility. Finding the right balance between those two contrasting requirements is the heart of the challenge. Kubernetes and Docker, as well as IaC tools like Terraform, Cloud Formation, and Azure ARM are a great substrate as they provide a consistent framework to manage applications across a wide range of infrastructure resources between clouds. 

The maturity and adoption of these tools remove the barrier needed to build a developer platform. This opens the opportunity for creating a custom platform per organization, allowing each team to have a high degree of control over how the platform can fit into their organization’s environments and resources.

This approach provides a much better balance between simplicity and flexibility because each organization can choose where to draw the line and even change that line over time without being dependent on an external vendor for every such change.

Introduction to Backstage

Backstage is an open source IDP project led by Spotify. It consists of three main parts:

• Service Definition – the service definition is based on the Kubernetes template format and provides a service descriptor that can fit into a wide range of services, such as a monitoring service, CI/CD platforms, Git repositories, etc.
• Backstage service catalog – basically a web application with a database backend that provides a single point of access to manage those resources
• Integrated tooling via plugins – provides a marketplace of resources integrated into the platform

The following screenshot illustrates how an IDP based on Backstage can look.

As you can see, we can use Backstage to organize all development services and resources such as our squad KPI, the development pipeline, API and Git repositories in one place.

Turning Backstage into a Full-blown PaaS

One of the missing pieces in Backstage is the infrastructure provisioning part, which is a critical part needed to turn Backstage into a full-blown PaaS.

Cloudify provides a remote execution engine that integrates with a variety of infrastructure resources and allows users to organize them into a self-service development or production environment.

The integration of Cloudify into Backstage includes synchronizing the development environments into the Backstage service catalog and allows self-provisioning of those services through the Backstage portal or through GitOps.

You can watch a demo or get more information on the integration from the Cloudify Backstage Git repo.

The End of ‘Growth at any Cost’

Earlier this year I wrote about the analysis by Sarah Wang and Martin on the impact of the cost of cloud on hyper growth companies. As tech companies trade velocity for efficiency, they find themselves in a non-sustainable growth model where the cloud costs surpass 50% of total revenue. Their post illustrates how this inefficiency directly impacts the company’s profitability—and valuation.

Their analysis was conducted prior to the current stock market fallout. An arguable point then, in today’s market dynamics, it would be less difficult to convince anyone that a “growth at any cost” policy is unsustainable. Today, DevOps teams have to find ways to support growth while controlling costs. This forces organizations to better control the ways development teams use cloud infrastructure. 

IDPs have become a major tool in this effort, and unsurprisingly they’re becoming the next hot DevOps trend. Spotify Backstage provides a great framework and ecosystem to shorten the time a team needs to build its own internal platform. Cloudify is the open, remote execution engine that provides a single place to organize infrastructure resources into a self-service environment. The integration between Cloudify and Backstage turns Backstage into a full-blown PaaS.

The current Cloudify-Backstage integration is only an MVP release. Next, we’ll deepen the  integration, providing more monitoring within the Backstage portal. You can find more information on our open source contribution to Backstage here. I would love to hear your feedback on where we should invest next.

Using Open Source Backstage and Cloudify to Solve the Speed Paradox 

The “speed paradox”—where the faster you grow the more fragmented and complex your software ecosystem becomes, which in turn slows everything down again—needs not to grind our DevOps agility goals into the ground. By using open source Cloudify integrated into Spotify’s Backstage project, DevOps teams can avoid becoming the new bottleneck in software development agility goals. 

References

• Use Monitoring Insights to Optimize Cost (Thenewstack)
• Innovation Insight for Internal Developer Portal (Gartner)
• DevOps, PaaS And Everything In Between (Blog)
• Cloudify 2022 Predictions: Bring on the next wave of DevOps (Podcast)
• IDP at Wix: Why We Need A God To Manage Our Infrastructure’ ft Matan Cohen, Software Group Manager & Tech Lead At Wix.Com (Podcast)
• IDP at AppsFlyer: How to Approach Platform Engineering?’ ft Roi Ezra from AppsFlyer (Podcast)
• Podcast | REVERSIM – The Trillion Dollar Paradox ft Martin Casado (Podcast)
• Reducing Cloud Spend Need Not Be a Paradox 
• The Cost of Cloud, a Trillion Dollar Paradox