Bridging the Gap Between Applications & Cloud Environments

August 30, 2022
Posted by: Nati Shalom
Category: Cloud, Cloud Automation, Cloud Deployment, Cloud Management, Cloud Orchestration, Cloudify, CMP, Configuration Management, Deployment, Developers, DevOps

The Story Behind the Cloudify 6.4 Release

Platform engineering teams are now being tasked to build internal development platforms to simplify the development process, specifically in the area of Kubernetes and Terraform-based environments. Building such an internal platform is still a complex task – as noted in the Dan Moore post on their journey with Terraform at FusionAuth. The main challenges are:

Need a lot more code
Need an SWE with cloud knowledge
No ‘one easy place to track infrastructure’

Cloudify 6.4 was designed to save a lot of that custom work by taking the common best practices and lessons learned from the “Spotify” of the world and productizing them into an open-source product offering so that everyone can implement them. This enables the platform engineering team to turn their existing Terraform modules and Kubernetes clusters into a self-service environment in a matter of hours. In addition, we added pre-deployment validation such as TFLint, tfsec, tagging, and cost estimation, as well as post-deployment validation as part of the environment definition. This way, we ensure correct and secure use of those environments across teams.

I’ll start with the industry perspective and gradually dive into the specific product features behind the release.

Join the upcoming Cloudify 6.4 Webinar to learn about all the new features and cap[abilities.

What is Platform Engineering?

A good description of the term can be found in this post: What Is Platform Engineering? The Concept Behind the Term

“Platform engineering is a process organizations can use to leverage their cloud platform as efficiently as possible so that engineers can deliver value to production quickly and reliably. It consists of removing obstacles between developers and production.”

How to Scale a DevOps Workflow?

Today, cloud developers only spend 11% of their time on average actually writing code, with 14-16 hours spent every week maintaining internal tooling, setting up dev environments, etc.

This gets worse at scale …

The solution is to abstract this complexity by turning your infrastructure resources into self-service environments.

A Three-Step Approach for Creating Your First Self-Service Environment in less than an Hour!

One of the key goals in Cloudify 6.4 was to put together all the features that are needed to turn an existing resource into a fully managed self-service environment in one place.

This includes a set of features that simplify the process of integrating and importing those existing resources as building blocks.

Creates a layer of consistency that will allow you to stitch those resources together under a common environment definition.
Allow sharing of information between resources and access control for accessing those resources across teams and projects.
Allows teams to continuously update those environments as needed.

The three steps approach aims to streamline that process end to end, as illustrated in the diagram below or in more detail with our demo.

Adding Pre and Post Deployment Governance as part of the Environment

Cloudify 6.4 includes integration with DevSecOps tools such as TFlint, Tfsec, Terratag, Terragrunt, and Infracost, which provides a rich set of pre-deployment validation. Additionally, Cloudify 6.4 includes integration with post-deployment policy management such as ServiceNow ITOM. This way, we can ensure that each environment is self-governed.

Cloudify 6.4 Highlighted Features

A Single Place to Manage All Your Environment Resources

The Cloudify Catalog is a repository that allows you to manage all your resources and environments in one place. A resource can be IaC resources based on Terraform, CloudFormation, Azure ARM, Kubernetes Clusters such as EKS, AKS, GKE, Configuration management such as Ansible, Docker images, and VM based resources, etc.

We added more than 130 standalone services of all kinds. This includes a set of certified environments that cover more complex use cases such as Multi-Region Kubernetes for DR purposes, as well as an out-of-the-box example of a cost-optimized development and production environment.

GitOps and Git Actions at the Environment Level

GitOps allows the operation team to streamline the development workflow i.e. pull, push, merge, etc., with the operational workflow init, plan, and apply. Having said that, this mapping is too open-ended and requires a lot of human responsibility to make sure that the right mapping is implemented, etc.

Applying GitOps at the environment level provides more structure to how this mapping is done. This also allows us to put a control point such as an approval process during that process to avoid triggering the wrong operation straight into our production. All this can be fairly implicit and would therefore simplify this process significantly with less exposure to human error.

You can see a demo of how this works.

Pre Deployment Validation

One of the key challenges in managing IaC-based infrastructure is that any misconfiguration can have a significant impact, as seen in the example below:

The most effective way to protect from such misconfiguration is to detect them as early as possible in the development process – this is also known as the Shift Left Approach.

Cloudify 6.4 comes with built-in support for a number of such validation tools that can now be integrated as part of the environment definition. Currently, this includes IaC configuration management such as Terragrunt, Syntax validation such as TFLint, static code analysis, security validation such as Tfsec, tagging validation through Terratags, and cost estimation analysis such as Infracost.

You can read about this feature in more detail – click here.

Simplifying the Continuous Update Operation through a New Intelligent Update Workflow

Each environment resource maintains its own update processes, which often makes the continuous update and day-2 operation across all those resources quite complex.

Cloudify uses its intent-based workflow mechanism as well as the plugin to provide a consistent update interface across all the underlying resources that are part of an environment.

Therefore a user only needs to update the desired state of its environment. Cloudify will calculate the delta between the intent and the current state and, in turn, will call the right update process for each resource. For example, it will call a Helm update when updating the Kubernetes Service or Terraform Apply on the Terraform infrastructure resource.

Users can also customize this process by implementing their own update operation on their custom set of resources.

Simplifying the Way Environment is Managed Across Teams and Organizations

A typical environment includes a set of secrets, configuration variables, modules, helm charts, ansible playbooks, git repositories, and scripts.

In addition, a typical environment is often organized in some sort of folder structure and Github repositories.

In reality, however, mapping those resources into an organization’s structure is non-trivial since, quite often, the same resource can be shared between teams. A certain set of resources may access as a reference only to other teams, while others may have provisioning rights.

Most users would only need to be exposed to the endpoint references of a Database or Kubernetes cluster, but the entire set of resources is part of that resource state file.

In addition to that configuration secret, they are also organized in a similar structure where some would have a global scope, some would be shared with certain teams, and some would be private to a particular project or resource.

Cloudify’s environment structure allows teams to organize secrets, configuration variables, modules, and other resources in their natural organization structure. This provides consistency for managing and mapping those resources across all infrastructure components.

Cloudify also allows dynamic matchmaking between deployments and environments i.e. the same deployment can be deployed to multiple environments based on tagging and filtering.

What’s Coming Up Next?

Our next release – Cloudify 7.0, will be a continuation of the journey that we set in Cloudify 6 with a specific focus on the following areas:

1. Speed up the time it takes to turn your existing resources into a self-service environment.

Streamline the three steps import -> compose -> update
Low Code <> No Code experience to create environments (a reference to Spotify C4 model)

2. Significantly simplify the continuous update process of environments by offloading many of the complexities involved with updating an environment infrastructure from the CI/CD pipeline to the environment layer. Since the environment layer comes with intimate knowledge about the resources and their relationship, it can take a more intelligent workflow decision to deal with cases of drift management, partial failure during an update, as well as choosing the right update method for each target resource.

3. More integration with IDP platforms such as Backstage. Similar to our work with ServiceNow as well as with other CI/CD platforms such as ArgoCD and Bitbucket.

4. Full SaaS – We are keeping to our strategy to take an Open Source first approach to our SaaS. Our goal is to allow users to download the product on their environment or have a fully SaaS-managed offering, as well as a SaaS-managed option that will allow users to manage their local Cloudify instance through our SaaS service. We’ll provide more specific details in a separate post.

Final Notes

For more information on Cloudify 6.4 refer to the new Introduction to Cloudify whitepaper, demos, and release notes.
To learn more about how these features are used to drive large-scale enterprise environments across Azure and VMware-based infrastructure, sign up for our Cloudify 6.4 webinar.

References

Take Cloudify for a Test Drive

Start your free trial Visit Our Documentation Join Our Slack Channel