Devops is a popular approach to the software development lifecycle that seeks to improve the quality and frequency of software releases. This article examines the practice of Devops, the role of automation, and the opportunities for unheard of efficiency in the delivery of software as a result of the virtualization revolution.
Devops And The Role Of Automation
The Emergence of Agile Methodologies: Traditional old school software development mimicked the design and production of physical things like buildings and aircraft. A blueprint or codified design was created on paper, a schedule was produced with milestones attached to the specification, and software teams implemented the designs. After implementation, the software was often handed over to a QA team whose responsibility it was to verify quality and compliance with the specifications. Finally the operations team would deploy and manage it. As noted in those days, not only was implementation success rare, but even if successful, the resulting system was unsatisfying to the end user (even if they approved the specification).
The central conceits of the failure of waterfall development was that:
- Requirements are fixed and known.
- End users know precisely what they want.
- It is possible to capture all the subtleties of a complex system including unknowns.
By the 1990s, iterative development approaches were being used to limit risk and increase flexibility. These attempts didn’t fundamentally change the process, but broke it down into shorter release cycles so that course corrections could be made. It was still a documentation heavy process, but helped lower the chance of bad surprises at delivery at the potential cost of efficiency.
In the 2000, Agile development was introduced with the publication of the Agile Manifesto. The Agile Manifesto isn’t a software development methodology, but rather a set of guiding principles for software development in the presence of partially or poorly understood requirements. It emphases adaptability to changing requirements, frequently releases, self guided teams, and eschews specifications and the related detailed scheduling.
The challenges encountered by those attempting the rapid release cycles advocated by Agile led directly to the modern day conception of Devops. The QA team is no more, largely automated away by automated testing. In the pursuit of ever higher quality and speed, automated testing is frequently triggered directly by submissions to software repositories. The “Ops” part of Devops is not the likewise replacement of operations teams, but generally refers to the automated release of software, ideally to production. The automation of this entire process, from software development through testing and production deployment defines Devops. Without extensive automation, the Agile approach bogs down in the testing and delivery phase which winds up consuming the lion’s share of time and resources.
Book a demo with Cloudify and understand the power of infrastructure automation!
“Infrastructure” refers to the platforms and networking environments that applications operate in. One of the central challenges of software testing (Agile or not) is the provisioning of realistic testing environments so that production releases aren’t events of high drama and terror. Infrastructure automation is the process of creating environments for the testing and operation of software. In the old days, this meant having a lab with production equipment in it (or as close as possible) that the operations team would construct for running user acceptance tests. Another option was (and still is) hiring a testing firm.
With the advent and widespread adoption of virtualization technologies, including cloud providers, the opportunities for creating meaningfully accurate test environments on the fly has never been greater. With the click of a button, complex environments can be created, used, and torn down either on premises, the cloud, or often in the case of hybrid use cases, both.
The promise of on-demand cloud infrastructure automation has inspired the creation of a number of infrastructure automation tools for the purpose, including cloud neutral tools like Cloudify and Terraform, and cloud specific infrastructure automation solutions like AWS CloudFormation, Azure ARM, and Google Cloud Composer.
The previous discussion of automated testing dealt with tests triggered directly by developer actions. These tests are typically limited “unit” tests that don’t really exercise the overall system in a realistic way. They typically use simulated interfaces (mocks) to permit testing in isolation. To pave the way for the Devops goal of creating production quality systems frequently, the ability to perform realistic “integration” testing, which tests the system as a whole. This is where the fusion of infrastructure automation and devops shines.
The integration of infrastructure automation and devops means that test environments of arbitrary complexity can be created and torn down in concert with testing needs. Even companies with limited cloud presence can benefit enormously from leveraging the scale of the cloud for testing purposes.
Beyond testing, for organizations (like most) that have a cloud (public and/or private) presence, the ability to create environments on demand in concert with software releases, means that true infrastructure as code (IaC) practices can be adhered to by versioning and releasing the production environment automation templates. The infrastructure orchestrator can respond to release events by updating server configurations, networking parameters, scaling up and down, and so forth.
Putting it all together, the addition of infrastructure automation to devops completes the Agile vision of small frequent releases, tightly coupled with developer activities, and thorough testing. It even folds infrastructure itself into the Agile process, making infrastructure versionable, repeatable, and flexible.
Cloudify And Devops Automation
Cloudify provides an industry leading portfolio of Devops integrations that can serve as tangible examples. These include integrations with Jenkins, Github Actions, and CircleCI. These integrations can be used both to construct test environments, or implement infrastructure as code by performing updates on existing environments.
A plugin for Jenkins enables commanding Cloudify managers to support any of the use cases discussed in this article. The plugin is available on the Jenkins plugin site. The plugin provides support for using Jenkins to command Cloudify to create and destroy environments for testing, for daisy chaining Cloudify blueprints to pass data in complex scenarios, and using Cloudify as bridge to other provisioning tools, to make common integration easy with tools like Ansible playbooks, Azure ARM templates, AWS CloudFormation templates, and Terraform modules. It even supports the on demand creation and destruction of Cloudify managers to support build processes.
Cloudify supplies a wide variety of Github Actions to support CI/CD operations using a Cloudify Manager. These range from environment creation and destruction, running arbitrary CLI commands, executing arbitrary Cloudify workflows, and automating infrastructure from popular providers including Azure ARM, AWS CloudFormation, Terraform modules, and GCP Kubernetes clusters. Environment outputs are made available to enable combining multiple Cloudify operations together, or enable integration with other tools. For example, an environment might acquire a dynamic IP address which is exposed in a blueprint’s outputs, which can then be passed to an action that performs tests against it.
Cloudify supplies a CircleCI Orb for commanding a Cloudify Manager. These commands include environment creation and destruction, running arbitrary CLI commands, executing Cloudify workflows, and creating environments on Azure ARM, AWS CloudFormation, Terraform, and GCP Kubernetes Clusters. As with the other integrations, there is a command to retrieve outputs and capabilities from Cloudify deployments/environments for informational or integration purposes.
The software development lifecycle had its initial concept rooted in processes for creating physical engineering products like buildings and airplanes. This model evolved over the years in response to increasing complexity and industry experience with development. The evolution progressed from a specification driven process to a more prototype driven process. Along with the process evolution, the hardware evolution provided the means to build and release very quickly.
This combination of an appetite for small, low risk releases, and the availability of fast hardware set the stage for Agile methodologies, which prevail today. Real world Agile teams rely heavily on automation to compress the old development pipeline into a continuous stream of releases to production.
In order to automate the “last mile” of Agile processes, you need Devops infrastructure automation. This permits both the construction and teardown of elaborate test environments, it also allows the operation environment itself to be treated as code and become part of Agile processes.