We started using Vagrant and Packer for our build process. You won’t believe what happened next…

There are good reasons for applying immutable infrastructure practices; one of which is that you don’t have to worry about your servers containing leftovers from previous deployments. Additionally, your deployment processes can be extremely clean as they don’t have to deal with the ins and outs of the live updates of middleware and code but rather the creation of images/containers and the infrastructure containing them.
At Cloudify, we have many artifacts to build. From Docker and machine images to potentially relocatable Python virtualenvs, node.js tar files and Windows binaries.
We want to our build process to be immutable.

Experience the awesomeness of the Cloudify Blueprint Composer today.  Go

Vagrant

As we’re not a SaaS company, but rather a provider of Open-Source software, we thought: Why not use the same concept of immutable infrastructure to build our artifacts? What that practically means is spinning up a new environment each time we want to create a binary. And what’s the easiest way to spin up a temporary environment? Vagrant.
Let’s say we want to create a Docker image. We create a `Vagrantfile` containing 2 VMs:

The first AWS VM is used in our official build process.
Thanks to Vagrant’s abstraction over the IaaS, we can choose the properties of the machine fitting a specific build which allows for easier optimization of the build process. For instance, if we know we have a long and complex build process for a specific artifact, we might want to provide lots of resources to the build machine (and vice versa).
By using Vagrant, we’re also able to provide locally executable build systems. The 2nd VM allows people to build whatever artifacts they need locally instead of being tied up to a specific build system.
The `provision.sh` script is simply the script required to generate the artifact. In this particular case, we install and configure Docker and docker-compose; run a `docker-compose build …` on a Dockerfile residing within the synced directory and save the image to a tar. Then, we push the tar to S3 automatically. `cloudify-packager` in the above example contains the files requires for the build process so they’re rsync-ed up automatically and used from within the VM.
Another example would be creating Cloudify agents for different distributions. We require compiling our agents on Ubuntu Trusty and Precise, CentOS 6.4, Debian Jessie and Windows Server 2013.
We have a `Vagrantfile` containing AWS and Virtualbox based VMs with images matching the OS and distributions we need. We supply a multi-distro supporting `provision.sh` file to build all Linux based agents and another `provision.windows.bat` file specifically for windows.
All that’s left then is to run `Vagrant up debian_jessie_aws` and poof!, a debian jessie agent is generated.

Decoupling Heaven

This process completely decouples each artifact’s generation mechanism from the specific build system managing it. It also allows us to parallelize our build process as we can spin up endless amounts of Vagrant machines in whatever IaaS provider we choose (as long as there’s a Vagrant plugin for that provider) and don’t have to rely on our Jenkins cluster’s capacity or on us writing logic for generating machines for the build process.
In addition, it doesn’t matter which build system we’re using to create the artifacts. It’s all just about executing the same command in different environments.
This decoupling mechanism also allows us to provide a way for our customers to generate artifacts on their own. Being Open-Source, a `git clone` and `Vagrant up` is all a customer needs to build their own artifact.

Readable Logs

Vagrant provides decently readable logs during its provisioning process. This allows us to relatively easily debug build related problems. In the future, we intend to ship these logs to an external ELK stack, to Logentries, or to Loggly to be able to analyze our build process in deeper granularity.

Where Packer comes in to play

Obviously, we’re not interested in installing all build requirements each time a VM is spun up.
We’re going to use Packer to build images containing the basic requirements for our build process.
We’ll use a packerfile.json file that contains configuration for creating an image containing Docker to be used by our Vagrantfile later on. (yes. We could use the Docker builder but as it runs on the local machine, and we’d rather spin up a new, clean VM, we do it this way.)

The `provision.sh`, in this case, installs Docker and `docker-compose` and exposes the Docker API (for docker-compose). The generated AMI is then used in the Vagrantfile. Another example would be generating our agent packages which require ruby, fpm and packman. We can use Packer to create an image containing them for each distro and.. fun!

There are cons, but even so…

This solution isn’t perfect. Spinning up VMs takes time and is a process prone to environmental failures. In addition, it costs more money and exposes some security related issues (obviously, security-wise, no-VMs is better than VMs).

Win!

What we’ve come to see is that it’s worth it. Before making the switch, unclean machines were generating our artifacts and we constantly stumbled upon failing builds. Adding several minutes to a parallel execution of multiple artifact build flows is not a deal breaker. The machines are not up long enough to be breached (and security-groups are configured properly of course); and the few hundreds of dollars a month are not an issue when this solution provides us with the ability to get a clean environment every time.
By the way, the time added to spin up VMs can also be reduced by building on clouds providing faster VM provisioning like local OpenStack installations, Exoscale (hosted Cloudstack) or Digital-Ocean. We chose AWS thanks to its stability and availability. We might (and probably will) switch to a different provider in the near future. Who knows… maybe we’ll start using Docker for that…

So?

Since we started building our artifacts as described above, we’ve mostly only experienced problems with build process logic and (rarely) missing prerequirements. These can be substantially reduced by using images which contain as many prerequirements as possible.