Cloudify and Infrastructure As Code

Introduction

Cloud automation is built on a foundation of virtualization, in which physical infrastructure can be simulated by software. This simulated hardware, unlike physical hardware, can be provisioned on demand.  This infrastructure as a Service (IaaS) provides great flexibility and efficiency, but at the same time makes a formerly static “box” into a programmable entity. The ability to spin up large numbers of virtual machines and related networking infrastructure (and other services) on demand brings with it a need for a concrete definition and traceability of the infrastructure and its various relationships over time.

Continuous deployment is a practice of automatically delivering software as the result of a build process. Combined with automated testing, the delivery of features to users is greatly sped up. “Infrastructure as Code” (IaC) is the understanding that the virtual infrastructure is software, and in concert with an orchestrator can participate in the same continuous deployment practice that other software does. This article examines Cloudify’s role as an enabler of IaC. Cloudify has been doing platform agnostic declarative infrastructure (IaC) and cloud provisioning since before the term existed.

Infrastructure Model

IaC requires a language for defining the desired infrastructure and its relationships. Cloudify uses a dialect of the OASIS TOSCA language to declaratively model infrastructure configurations. This language is itself extensible, allowing the development of vocabularies that are purpose built for different target platforms. The models (blueprints) in Cloudify are editable text files well suited for versioning in git or other version control system as “code”. Version labels can be applied to blueprints upon upload to Cloudify’s internal repository for traceability. 

Cloudify’s extensible modeling language (DSL) enables the abstraction of frequently complex tools in a universal abstraction layer. Modeling language extensions (plugins) enable the representation of toolchain components at any level of abstraction desired. Included plugins enable cloud automation tools like Ansible automation, Terraform automation, shell scripting, and others so you can reuse existing automation investments, and future proof for the inevitable new tools of the future.

Infrastructure building blocks are made available to Cloudify blueprints that support simple and complex infrastructure models. Models/blueprints using the building blocks can encompass multiple infrastructure providers simultaneously, supporting use cases like hybrid cloud. A wide variety of infrastructure plugins enable blueprints to support infrastructure from well known providers:

• Amazon Web Services (EC2)
• Amazon Elastic Kubernetes Service (EKS)
• Azure ARM
• Azure Kubernetes Service (AKS)
• Google Cloud Platform(GCP)
• Google Kubernetes Engine (GKE)
• Openstack
• OpenShift
• VMWare NSX-T
• VSphere

To support IaC, having a static model will not suffice. A key concept underpinning IaC is the idea that incremental change is critical to the process. Changes to the model drive CI/CD processes that culminate in changes in the virtual infrastructure. A change to add a server or a network to the model cannot be achieved by tearing down and reconstructing it. Cloudify’s deployment update feature supports this use case by automatically identifying and rendering changes in the target infrastructure.  Also supporting model evolution is the Cloudify feature called service composition. In a complex infrastructure model, it is often useful to segregate the model based on individual components’ lifecycles. Service composition allows you to decouple architectural components that have distinct life cycles or management needs.

Practically speaking, especially for large scale infrastructure, the ability to retry and resume in case of failure is essential. Cloudify’s execution model uses a retry mechanism to enable recovery from recoverable errors, and it has the ability to restart failed processes when they can be fixed by external actions. For example, imagine automating the installation of 500 VMs and the process fails due to quota consumption. The resumable workflow capability of Cloudify enables the quota to be adjusted and progress resumed, without requiring a complete reversal and redeployment.

Another key feature of any IaC orchestrator is secrets management, which shouldn’t not be stored in version control or visible in templates. Cloudify provides a built in secret store that integrates seamlessly with models, allowing secrets to be referenced by name while keeping them safely encrypted.

CI/CD Support

Closely related to, but independent of, the IaC concept is continuous integration and delivery (CI/CD). CI/CD concerns the automated testing and deployment of software. IaC essentially turns hardware into software, making it a natural fit for CI/CD. One aspect is supporting the testing of applications by constructing test environments on demand. The test environment becomes another versioned code artifact in the automated process, enabling well defined, unambiguous integration tests. On the delivery (CD) side, production infrastructure can be defined in blueprints in version control, where releases can trigger automated infrastructure changes. For example, update a blueprint in git, and as a result automatically add a server and update a load balancer configuration. When such a connection is established between version control and infrastructure, the true power of IaC is realized.

Cloudify has integrations to popular DevOps deployment tools that play key roles in the CI/CD/IaC processes. The Jenkins integration connects build jobs with the Cloudify manager, running workflows that perform installation and updating of infrastructure (and applications). Custom and standard environments are available such as Kubernetes, Azure ARM, Cloudformation stacks, and others. For version control support, Cloudify provides Github Actions in the Github Marketplace. These actions provide a wide array of capabilities from basic CLI commands to environment creation and workflow execution. For users of the CircleCI system, there is a Cloudify Orb that provides similar functionality as the Github actions, but for CircleCI pipelines.

Other Northbound Integrations

The CI/CD tool support described above represents a northbound (from Cloudify’s perspective) integration. For a system to fit well into an existing ecosystem of tools, other integrations can be of value.  Cloudify provides several “northbound” features that make it play nice in your tool chain:

API

Cloudify features an extensive REST API for seamless integration with operational environments. The API supports the complete set of commands to deploy and operate (run workflows against) blueprints, as well as manage the platform itself. A corresponding Python SDK is also available in addition to a command line interface.

ServiceNow

For those wishing to implement ITSM processes along with IaC and CI/CD, Cloudify supplies a ServiceNow Application. The ServiceNow Application enables the construction of flows that automate IT infrastructure deployment in combination with change management processes. A common use case is the definition of approval gates that control the modification of critical infrastructure.  In this way, engineering is (potentially) decoupled from deployment decision by the formal approval process.

Authentication/SSO

Cloudify supports both LDAP and OKTA support as well as a general extension framework for authenticators. Integrated user account management reduces complexity and increases security in any integration effort, which multi/hybrid cloud IaC by necessity always is.

Infoblox/IPAM

Cloudify has an Infoblox integration that enables infrastructure deployments to benefit from IPAM functionality.

Beyond Infrastructure

While the main topic of this article is Infrastructure As Code, it has to be recognized that automated infrastructure is a means to an end: the delivery of value to end users. If your users need services as well as infrastructure, a well designed platform can serve both needs. After all, if infrastructure is code now, code is certainly code and the IaC platform should be able to support both. As mentioned above, there is an intimate relationship between CI/CD and IaC in that CI/CD can release, manage, and deploy infrastructure templates, and IaC can support continuous integration by constructing repeatable, consistent test environments.

Cloudify’s base modeling language doesn’t distinguish between infrastructure and services, and provides an “environments” abstraction that essentially designates infrastructure models that can host services. Services can be modeled in Cloudify’s modeling language, including any lifecycle management logic. The integration with tools like Ansible and others make it compatible with existing automations, while upgrading them to a declarative approach. Service models in cloudify are not just static descriptions of service configurations, they are runtime-active models that can store state and support complex workflows for provisions through day-2 operations.

By supporting both IaC and service lifecycle management, Cloudify can be a key part of a toolchain that enables:

• Definition, versioning, and provisioning of complex test environments
• Definition, versioning, provisioning, and test execution on the test environments.
• Scalable infrastructure models that can support needs from development through product.
• Highly available orchestration of UAT and production environments which support day 2 modifications.
• Highly available model driven orchestration of service life cycles that support day 2 operations.
• End to end service orchestration.
• Service composition and chaining.
• Coordinated infrastructure and service automation.

Summary

The infrastructure as code concept is merely the formalization of the recognition that in the cloud era, infrastructure is already code. While trivially, the code could literally be code (i.e. scripts), the purest realization is via declarative modelling. A CI/CD pipeline needs a DevOps automation platform to push those models forward. All the major cloud/container platforms have their own proprietary models and orchestrators (and APIs), as well as CI/CD systems. Obviously this leads to a form of lock in, and prevents both flexibility and best of breed implementations. An unopinionated, future proof, declarative orchestrator like Cloudify with robust north and southbound integrations can meet the needs of multi-cloud IaC, and go beyond into the services orchestration realm.