Deployment Automation, Cloud Management & Monitoring with Puppet & Cloudify

Ok, so our new configuration is great, but now it’s time to think about taking our cloud automation (see previous post) to the next level. While Puppet takes care of our deployment phase, we are still stuck with manual provisioning of the machine and manual management of the post-deployment phase, such as self-healing, automatic scaling and more. But not to worry too much – Puppet partnered with  Cloudify helps simplify this significantly.
Cloudify automates the provisioning of the machines for us, really for virtually any cloud (AWS, Rackspace, OpenStack, Cloudstack, vCloud, {insert preferred cloud here}), even if you work in a private data center with physical machines.

Once Cloudify instructs the cloud to provision the virtual machines, it can then install a Puppet Agent for us, after which the Puppet Agent will continue the installation step.
Cloudify works with application orchestration plans called recipes. However, there are a few differences – since Cloudify takes care of pre and post-deployment, the recipe can also include the service dependencies, type of machine for each service, as well as, monitoring and scaling configuration. It is also possible to extend another recipe by overriding specific parts of the lifecycle, and changing pre-defined properties, and connecting to other remote services and third-party apps such as monitoring apps (Zabbix, Nagios, Logstash, Zenoss etc.).
Now we can try and take it to the next level … let’s see what that looks like:

The Flow
1. Cloudify provisions a new machine.
2. It then installs a Puppet Agent (via the Cloudify Agents).
3. The Puppet Agent then connects to Puppet Master.
4. The Puppet Master will provide the required module.
The recipes are pretty simple, all they require is for you to define:

• The service name (so it can be translated by the Puppet Master to the right manifest)
• The specifications of the app lifecycle, such as start and stop detection to know when the service is up and running
• Finally the scaling rules and any other actions in the post-deployment phase

Please note, that the Puppet Master can be deployed automatically (via Cloudify) or manually (and then you need to provide the Puppet Master IP). You’ll likely already have a Puppet Master deployed (and have some modules and manifests installed), so in that case, you can use those.

Challenges – Classifier and Signing to the Puppet Master
The main question is how does the Puppet Master know which module to provide? Since we are working in a cloud environment, the host name of the new machine is meaningless. We need another way to let the master know that we need a specific service, say MySQL, for example. This is done using the Cloudify/Puppet integration which leverages the Puppet facter. Cloudify can use the facter to send facts to the classifier, a classifier which presents the required module instead of relying on the node name. We add another section to the site.pp code that will analyze the Cloudify classifier variable, and based on the value received, will determine which modules we want to include and install.

If you take a closer look at the integration, you will find under the Puppet recipe directory, PuppetBootstrap.groovy, which is invoked as part of the lifecycle managed by puppet-service.groovy. In the first file, you can find methods that are responsible for the bootstrap process as the name indicates, among them the creation of puppet.conf file using groovy templates. This file is also responsible for the interaction with the Puppet Master server in case an external classifier plugin is used.

Another issue you need to bear in mind – the interaction between the agent and the master requires signing. The agent signs to the Puppet Master and then the Puppet Master admin needs to approve this request to issue a certificate to the agent. This, of course, can’t work in our environment either, so the way to do this is to approve all requests by auto-signing and relying on firewalls and security groups to make sure that each agent is really allowed to access that Puppet Master.
Advantages of this Method

1. Automating Puppet deployment – agent installation
2. Automating the mapping of Puppet modules to the machines using the classifier
3. Dependency management (for example Tomcat depends on MySQL)
4. Overall monitoring – deployment process and post deployment state
5. Multi-Cloud – abstraction layer from the cloud allows us to use the same process on different clouds, also on BYON (bare-metal cloud) and on our local cloud.

Step by Step
Assuming we are working on our MySQL module, the MySQL recipe would look as follows:
1. We don’t need to replace or delete existing parts of the site.pp file, just to add something as the following:

2. In the Puppet Master add auto sign in for all agents – create the file under puppet/autosign.conf. – Add only * in autosign.conf.

3. One file would be the service properties file which should contain the Puppet Master server IP:
puppetMasterIp=”ip-10-4-30-136.ec2.internal”
Make sure that you provide this value to the puppet agent properties file.

4. The groovy recipe would look as follows (extending Puppet Agent):

So, whether installing your first MySQL with Puppet in just a couple of minutes, or taking it to the next level of cloud automation, as demonstrated in the previous post – Cloudify and Puppet integrate seamlessly through all the phases of application deployment to the cloud. Learn more about other configuration management integrations, and cloud orchestration tweaks.