Taking Jenkins CI from Automation to Orchestration – A Continuous Integration A/B Testing Use Case
The benefits of continuous integration orchestration tools are clear. The need to shorten deployment cycles that typically take six months to a year long is clearly evident. As such, focus on Jenkins automation is allowing companies to deploy code numerous times per day. With this massive amount of daily deployments there is a growing need to take the Jenkins orchestration portion of the process to the next level while ensuring it remains extensible, scalable, and easy to monitor.
Continuous integration orchestration tools have become more complex. Jenkins CI is designed for implementations requiring support of multiple use-cases and scenarios. Some examples include the building a robust CI pipeline all the way through continuous delivery, while others need support for A/B testing of deployments before committing to production. Many scenarios like this play out daily within those development teams using the Jenkins orchestration tool.
This is where Cloudify comes in to compliment and integrate with Jenkins CI. This post will dive into a more complex use case that enables usage of the entire CI process using Jenkins orchestration – all while A/B testing and monitoring are being completed throughout the process. In the end, we will see how to choose which code you’d like to deploy to production.
Want to learn more about Cloudify and Jenkins?
A/B Testing with Jenkins & Cloudify
The architecture we chose for this demo includes three environments, two that will represent testing, and one for the production environment. We will call them A/B test 1, A/B test 2, and Production for this demonstration of the integration with Jenkins CI.
Here’s an image of the environment:
To start with, we needed to create a single Cloudify main or primary manager as the single point of control for the three environments. We then used Cloudify Manager as a control-point for each environment. These managers, along with a Jenkins server, are what is needed in order to make up the suite of continuous integration orchestration tools for our example.
(Note: Cloudify’s RND actually uses Travis and CircleCI for such procedures. This same scenario can be run with most any CI tool such as QuickBuild, TeamCity, etc.)
Since this was actually the first time we attempted to deploy a Cloudify Manager with a Cloudify Manager, we were glad to see such success. It seemed that ‘drinking our own merlot’ worked with some tweaking. By doing automation this way, each Cloudify Manager inside the “main” manager can control various deployments independent of any of the other managers used during the orchestration.
For the purpose of this use case, we had to slightly modify the Cloudify Manager blueprint in order to disable Elasticsearch’s multicast. This feature is a part of Cloudify’s manager’s stack. This was necessary in order to prevent the communication/synchronization between the Cloudify Managers which all reside in the same network or subnet. Why is this useful? Because we actually need the disparity of these environments as that is actually an important factor in this case.
Here’s the modified manager blueprint’s snippet which disables multicast:
This deployment was run on Softlayer (IBM’s cloud provider). We took a basic Softlayer blueprint and bootstrapped (launched) the main manager. We then deployed the three different Cloudify environments by deploying three managers within the main Cloudify manager. Including one for the environment A/B test 1, one for A/B test 2, and a third for production.
Next, we created a Jenkins CI blueprint, which we then configured to start with three initial Jenkins CI jobs:
Two of these jobs were scheduled with Jenkins orchestration to sample a GitHub repository for the testing environments while one job remained ready to be run for delivery to the production environment.
Here’s an image that shows the Jenkins role in this scenario:
How it Works
This scenario works by sampling a Github repository every 15 minutes to detect and deploy any changes made. These changes are then pushed to the varying test environments. In our case this would mean A/B 1st test and A/B 2nd test, respectively.
When a change is detected in the repository, the first job will use Jenkins automation to deploy it on the first Cloudify manager. The second job will deploy on the second Cloudify manager. For each code push to GitHub, two jobs will be performed by Jenkins CI simultaneously. Each job will spawn its own independent deployment on its corresponding Cloudify manager.
The third environment behaves slightly differently. This is done intentionally, and does not actually run any commands in order to maintain the production job as semi-automated. This is necessary since it is essentially intended to be an A/B testing scenario. Having the manual intervention enables you to compare the two different testing environments and choose the version that is deployed to production.
Once an admin decides on the preferred version, they select a button that invokes the Cloudify workflow. The workflow then accesses Jenkins through the Cloudify Main Manager. Doing so will use Jenkins automation to move the selected version from either A/B 1st test or A/B 2nd test the rest of the way to production.
So while it is true that we are left with two testing environments containing minor differences, the semi-automated production deployment can still be fully scripted. This essentially closes that loop from continuous integration to continuous delivery.
In the Jenkins CI blueprint, we also created a custom workflow that allows users to run any Jenkins CLI command within the Cloudify CLI. Even though it is possible, you don’t even have to access the Jenkins CLI to run commands. This is a great example of how adding Cloudify to your box of continuous integration orchestration tools can be utilized to create custom workflow automation that remotely manages your environments without the need to SSH or other agents.
In this case we chose our application to be Drupal which is one of the most popular CMS platforms. We designed it so that every deployment on each one of the Cloudify managers consists of three tiers: Apache, MySQL and Memdached.
Here’s an image that shows how the Drupal application is deployed by Jenkins:Jenkins CI |
This example also demonstrates how you can leverage all of the added Cloudify capabilities that come out of the box to extend Jenkins automation and deliver robust CI through CD. Including everything required for post-deployment which is a critical element in the continuous delivery processes. This means you can essentially monitor Jenkins, the deployed application, as well as the entire lifecycle of the build and deployment processes. Items like success/failure and defined custom metrics based on your deployment are accessible through Cloudify’s policy engine.
This enables you to have an intelligent analysis of your system, however it was built. You can make decisions based on log files and other KPIs, receive alerts, and even have your system auto-heal and auto-scale based on these policies. Additionally, being infrastructure agnostic makes it possible to run this example on virtually any infrastructure like VM, bare metal server, Docker containers, or even a combination of them.
Another important element in continuous integration is cleanup so that you are not leaving a trail of unutilized machines behind you. This can also be scripted into your blueprint, enabling you to then undeploy each Jenkins CI job and the jobs or actual deployments spawned from it as well as its corresponding Cloudify manager. Other cool tweaks within the integration enable versioning with a manual cleanup process. This makes it possible to version deployments, and then choose a specific version to move to production at any time.
There’s plenty you can do with Cloudify combined with Jenkins CI as your continuous integration orchestration tools. In this article, I only just started hacking some of the “nifty” capabilities. For more information on Cloudify and Jenkins orchestration, you can watch the video in action below. Feel free to comment and share any other ideas you may have for getting the most out of your CI.
A trove of resources on this and other topics are available via whitepaper. This entire implementation, including an example Jenkins CI blueprint, is also available in Github.
Watch the video of this in action below:
