Pack Up the Wagon, We're Going Offline

This post was originally published on InfoQ.com in Jan 2016.

We’re a Python shop at Cloudify and we needed to be able to ship our plugins to customers. The plugins are sets of multiple Python packages which had to be available for different platforms, in different versions, for installation by our framework. The fact that we work with enterprise companies – banks and the likes – organizations that cannot simply install things off of the internet, is forcing us to provide the entire package and its dependencies available to the internet-less user. “pip install and be done with it” is not an option. This had us searching for solutions.
As such, during my work, I’ve had the distinct pleasure (note the sarcastic tone) of dealing with packaging artifacts, namely, Python packages along with their dependencies. These packages had to be available for installing at runtime and had to contain information on the packages for the installer to read.

Changing the face of DevOps. Cloudify glues it all together.  Go

Python doesn’t make handling dependencies easy. While you can statically link Python and ship it with dependencies, clever dependency handling is not Python’s strong side and it would require a lot of work to achieve a high level of portability although, as you will see, it’s gotten better. On top of that, Python’s default method of installing dependencies requires an internet connection which, even today, is not always a viable prerequisite.

Alternatives

At first, we thought about providing a PyPI (Python Package Index) clone along with our product. Unfortunately, PyPI is about 150GB and that is not something we can afford to ship to customers.
We needed a REAL offline solution, so we built one.
We have a lot of experience with virtualenv and Spotify’s dh-virtualenv came to mind. We could use it to package an environment with dependencies. Unfortunately, dh-virtualenv revolves around Debian packages and we needed packages for potentially any distribution. Additionally, our agents are already based on virtualenv and we just needed a way to install more plugins into the same environment.
Twitter’s PEX also came to mind. It is a way to create portable Python environments and ship them easily as single binaries. Unfortunately, after deploying such packages, we would not be able to install additional plugins in runtime, which is a requirement in our case.

Our Solution

What we came up with is Wagon – a very straight-forward solution for creating and installing a Python package. What it does is provide an abstraction for creating an archive which contains a Python package with its dependencies from multiple sources (GitHub, local path, PyPI); allows installing the package directly from the archive, validating it and displaying its metadata.
Please see the rest of the article at InfoQ.