Cloud Management & Usage Monitoring Through Continuous Integration with Quickbuild and Mist.io
I’m taking a break from blogging about my various Openstack related shenanigans. This time I want to talk about monitoring cloud usage.
I work with IaaS clouds – lots of them. The list includes several different AWS accounts and several HP cloud accounts. It also includes occasional ‘guest’ accounts on Softlayer as well as private Openstack and Cloudstack accounts. With so many resources, and so many projects, stuff gets dropped once in a while. VMs are left running, forgotten by the developer that was using them or missed by the automation that was supposed to shut them down. And in the cloud, if you forget something, you pay for it. Literally. VMs are billed for the time they are active, whether or not you were actually doing something useful with them. And before you know it, the bills start mounting, and the people from accounting show up at my desk.
So I need some way to monitor my cloud usage across multiple clouds and accounts. I spent a while looking online for a good cloud monitoring solution that would give me a quick overview of my current cloud usage. I love dashboards, by the way. Being able to see everything that interests me in one place makes things so much easier.
Cloud management & monitoring on any cloud with Cloudify. Go
After looking around at a bunch of cloud monitoring options, I finally found mist.io. It had just the right features I needed, and all that was left was to plug it in to my existing dashboard system.
So let’s start by reviewing the pieces.
Mist.io
Mist.io is a cool open-source project that monitors VM usage
across multiple clouds. The list of supported cloud providers is pretty extensive (see the up to date list here, that at this time includes:
EC2, Rackspace, Openstack, Linode, Google Compute Engine, SoftLayer, Digital Ocean, Nephoscale, HP Cloud, Bare metal servers, Docker containers and KVM hypervisors.
There is also a mist.io website, which basically offers the same product with some additional premium additions. I’ll be using this service for my purposes, though you can always install the open source version locally. Mist.io also offers a Python SDK which makes it a very scriptable system – I’ll be using the client SDK shortly.
Mist.io includes a console for your current cloud usage which is pretty useful by itself:
But for my project I also need to maintain history and statistics about my usage.
Quickbuild
The continuous integration (CI) system we use for Cloudify is Quickbuild. We’ve been using it for a very long time and it has proven to be a rock-solid system. Quickbuild also has a flexible dashboard system, where I can plug in my own custom data sources.
While this post deals a lot with the specifics of Quickbuild, the principles should be pretty much the same for any CI/Automation system.
The glue
The general idea is to define a configuration in Quickbuild that will poll the mist.io API for running VMs, collect historical data and display the latest results on a Quickbuild dashboard.
So let’s get to work!
Mist.io account
First up, you’ll need to set up your account on https://mist.io . It’s free.
Cloud credentials
For each cloud account that you want to monitor, create a dedicated user and give it the minimal permissions required to view the currently running instances. This is a good best practice for all integrations – use a dedicated account for each integration, using something like Openstack keystone or AWS IAM, and give it only the permissions it MUST have.
Configure mist.io backends
In mist.io, a backend is a monitorable target that can host compute instances. These can be Openstack tenants and AWS regions, for instance. You will need to set up a backend for each one of these. If, like me, you work across a lot of AWS regions and Openstack tenants, this part can get a bit tedious. So I wrote a couple of scripts using the mist client SDK to speed things up a bit.
Set up HP Cloud Backends:
Set up AWS Backends:
Just grab the scripts and tweak the credentials and tenants to match what you need.
BTW: I do wish mist.io would make this bit a little easier. There really should be an easier way to just give them the credentials and say ‘Monitor everything’
Create a script to collect the current compute instance details
I wrote another quick script using the mist SDK to do this. The whole thing is on github:
https://github.com/barakm/mist-monitor
This is the interesting bit:
https://github.com/barakm/mist-monitor/blob/master/mist_monitor/mist_monitor.py
Note how the python script generates an XML file with the details of the compute instances. Quickbuild likes XML files as input.
Configure Quickbuild
First thing we need to do is set up Quickbuild to accept the XML file format our script generates. This is a one-time operation, but you need to be an administrator of the Quickbuild server to be able to do this.
- As a Quickbuild administrator, go to Administration -> Plugin Management -> Custom Statistics Report -> Configure
- Click “Add New Category”
- Give your category a name, like “Running Cloud Instances” and an appropriate description
- Add two ‘indicators’ (fields in the report)
- running – This gives us the total number of running machines, and is usually the most interesting value. Set the XPath expression to – count(//machine[@state=’running’]).
- all – This gives us the total number of machines, so it includes machines that are shut-down or deleted. Not as interesting, but can be useful. Set the XPath expression to – count(//machine)
It should look something like this:
Set up the recurring task
With the custom Quickbuild category all set up, we can create the task that actually polls the mist.io API. This can and should be done with a regular Quickbuild user, not an administrator.
I have made the configuration available as a gist. You can import it from: https://gist.github.com/barakm/3927cc0e8930b259c69e
Or you can create it manually using the following instructions:
- Create a new Quickbuild configuration somewhere in the build tree. I called mine ‘CloudNodeMonitor’
- In the configuration definitions screen, go to Settings->Repositories
- Click the ‘+’ icon to add a new repository and choose a git repo
- Set the git repository URL to
https://github.com/barakm/mist-monitor.git
(you can always fork this repo if you want to add something). Make sure to give your Quickbuild repository a name you will remember. - In the Quickbuild configuration screen, go to settings->steps
- Add a new step (it’s the ‘+’ icon) and choose repository->checkout.
- In the step editing screen, make sure to choose the repository you created previously
- Add a new step, and choose build->shell/batch command
- Set the command field to:
./mist_monitor_runner.sh ${vars.getValue(“mistUsername”)} ${vars.getValue(“mistPassword”)}
Note how we are passing the mist.io credentials as Quickbuild variables – we will configure them later. - Set the Working Directory field to: mist_monitor
- Add a new step, and choose Publish -> Custom Statistics Report
- Set the Statistics Category to the name of the custom statistics category you created previously (something like “Running Cloud Instances”)
- Set the Files to Process field to:
mist_monitor/output.xml - Set the Report Set Name to: All_Machines
- In the Configuration editing screen, go to Settings->Variables
- Add a new variable. Call it mistUsername, and set its value to the username of your mist.io account.
- Add a new variable. Name it mistPassword (you may want to set the value to be displayed as a secret value, not a cleartext one) and set its value to the password of your mist.io account.
- Set the task execution schedule. Go to: Settings->General Settings->Edit and schedule the periodic execution of the task. Once an hour works out fine for me.
Your new Quickbuild configuration should look something like this:
Run the task a couple of times from the Quickbuild console to make sure it works as expected. Have a look at the ‘Latest Build’ tab to see the results.
Set up the dashboard widget
- Quickbuild has a built-in dashboard system that is pretty straightforward.
- Choose the dashboard you want to use (or create a new one)
- Select Add Gadget -> Others -> Custom Statistics
- Choose a relevant title and set the configuration to the task you created
- Set the Build field to: Latest Successful Build
- Set the Category Name field to the Custom Category you created (“Running Cloud Instances”)
- Select the “All_Machines” Report set and click Save.
You should see the latest results from your cloud monitor show up on the dashboard.
Click ‘View Report’ and choose the ‘Statistics’ tab, and you can see statistics on your cloud usage:
Now you have yourself a Quickbuild dashboard showing you the number of compute instances running on all of your clouds, courtesy of mist.io, plus some nice historical data as well.