Package Management. Packman Style.
We’ve been speaking quite a bit about the soon to be released Cloudify 3.0 – which is really a complete rearchitecture and rewrite of Cloudify from scratch, which brings with it all kinds goodness, but with that…some interesting challenges, as well. As part of the move to our new, and much more complex application stack, which is comprised of diverse technologies {namely – OpenJDK, logstash, elasticsearch, RabbitMQ, Riemann, Gunicorn, NGinx, Graphite, NodeJS, more than several Python modules… and more}, we found ourselves with automation requirements that weren’t quite standard.
To make the stack above available for offline installation, with integrated template-based installation scripts, template-generated configuration files and an easy to use package, to name the obvious, we realized we needed packaging software. We explored some of the existing packaging software and came up with a few options – OpsCode Omnibus, and Jordan Sissel’s FPM, are just a few – but our requirements weren’t exactly ordinary…so we got digging.
Let’s start with Omnibus. OpsCode created a packaging framework in-house called Omnibus to package their Chef installations and expanded it further to support general packaging of applications. When we started playing around with it, hoping to leverage it for Cloudify, we found we needed something with some different qualities.
- Something that would be able to retrieve resources from ANY type of repo (apt, yum, rubygems, pypi, etc…).
- We wanted a very easy to use “dict” – as config.
- We wanted something that will automatically generate config files and bootstrap scripts from templates.
So we moved onto FPM. This creates packages from directories, debs, RPM’s, Python modules and gems.. but..
- We needed something that would also download/install dependencies for Ruby gems and Python modules.
- We needed something that would handle yum and apt dependencies (do you know how many apt dependencies openJDK has?!)
- We needed something that would let us configure our entire application stack in one place.
- We needed something that would also provide us with a framework, not just a CLI.
- And if it could make us coffee too – that would be great.
And that’s when we decided to write Packman. Packman, how should I put this…creates packages. I guess it’s just that simple. Literally any kind of package. You can write a dict containing your package’s configuration and Packman will retrieve the resources and create a package accordingly.
How Packman Works
Packman uses run of the mill JSON to configure your entire stack. It has a built-in, neat and easy to use CLI tool called “PKM”, which provides access to basic features like:
- sources retrieval (‘pkm get -c COMPONENT’)
- packaging (‘pkm pack -c COMPONENT’)
- (or both – ‘pkm make -c COMPONENT’)
- Providing explicit dict files, component lists and exclusion lists –
‘pkm get -c COMPONENT1,COMPONENT2 -x EXCLUDED_COMPONENT1,… -f COMPONENTS_FILE
Additionally, it provides a way to override the basic implementation of the retrieval and packaging methods by providing external Python methods to performs these actions, and even allows you to combine the basic one’s and the added ones.
Some added bonuses:
- Template parameters are stated within the config, so there’s always only ONE file you need to manage when configuring the stack.
- A Vagrantfile is provided to run Packman and experiment with it (Ubuntu and CentOS are supported).
- And soon… much more (It’s still in Alpha..)
You can check it out here: https://github.com/cloudify-cosmo/packman and the docs are here – http://packman.readthedocs.org/en/latest/.
And definitely be sure to let us know what you think, and if you have any suggestions.
On the same note, we’ll be heading to DevOps Days Amsterdam this week – which we’re quite excited about, where we’ll be talking about packman and “log-aware-development”, and will be holding a Cloudify workshop – where we’ll show you all about cloud orchestration in action.