Remote Commands, Deployment Automation & SSH With Fabric & Cloudify

As more and more workloads and critical applications move to the cloud, remote execution is becoming even more important. Being able to connect to various remote elements is a key part of using Cloudify. In this article, we will explore remote execution in detail: why it’s needed, how to make use of it, and how it all relates to the Fabric plugin.

What is Remote Execution?

Remote execution is more or less how it sounds – it’s the ability to invoke commands on a remote server without dependency upon agents, assuming basic server connectivity is provided (such as SSH servers on Linux, or WinRM on Windows).

In this article, we will primarily focus on Fabric for SSH remote execution. We’ll run through some examples of usage, but bear in mind, this is not the only tool available for this approach. So, let’s start by running through some of the options available for remote execution.

There are many tools available for remote execution above SSH. Here are some of the most common:

• Linux
  • Fabric – written in Python
  • Capistrano – Ruby equivalent
  • Ansible – which is also in Python, but has more complex abstraction than Fabric including plugins and other niceties and provides some level of idempotency
  • SaltStack
  • Windows: You can also install SSH server on Windows and then connect via SSH, however WinRM has become the more common way to do this.
  • Ansible –  provides an abstraction above pywinrm

[This post on  WinRM vs. SSH will also help give some context.]

Generally, Fabric and Capistrano are referred to relative to code deployment, and Ansible is typically referred to around code deployment and configuration management.

A bit of a side note: there is an informal and debatable boundary that exists between configuration management tools (Chef, Puppet, etc.) and remote executors when it comes to code deployment and CM, but we won’t touch on that topic here.

When it comes to remote execution, there is certainly no shortage of opinions about approaches, so I find it best to address these first. As with any technology, there are upsides and downsides which can strongly influence your choices. Upon understanding the pros and cons for remote execution, you’ll be able to make choices about whether to use this method for configuring and maintaining servers, or code deployment. The alternative involves two methods: manual involvement along with config management tools, dependent upon the use case. Personally, I believe configuration management tools are not best suited for deploying code, but are a great choice for maintaining configurations.

Why You’d Want to Use Remote Execution

There’s no need to install agents.

An important aspect, as agent installation leads to additional overhead: managing the agent’s lifecycle, installation, and maintaining and monitoring its overall state

Security

Installing agents also requires the deployment of files and processes on your server – not all organizations are willing or able to support this approach. 

Rapid Changes

If your server is fresh out of the box and you don’t want to have to deal with the many tasks involved in configuration and setup, you can simply write a Fabric task and run it on-server.

(This does not account for the method of baking images, but we won’t cover that in this article.)

It Does Not (Necessarily) Depend on External Resources

Using an agent (for example, a Salt minion) requires the minion to be installed and configured on the server, then verified that it’s running. Of course it can be baked into an image or the installation and config processes fully automated, but generally speaking, this provides more moving parts to track and external resources to download, which are not otherwise prerequisites of your application.

Single Point of Failure

A Master-Agent based system requires a lot of work to prevent a single point of failure. A good example would be the relation between Puppet Master or Chef Server to Masterless Puppet and Chef Solo, respectively. Remote execution allows you to substantially reduce this risk, because you are not dependent upon a master server to perform tasks and operations.

No Server Utilization Overhead

Depending upon their implementation, agents may incur some overhead (CPU, memory, I/O, etc.).

Where Remote Execution Falls Short

Security

As a practical example, remote execution on Linux requires an open SSH port. Not all organizations are willing to leave this port exposed.

Idempotency

The execution environment is not necessarily built for idempotency. Ansible, for example, provides a certain level of idempotency. Fabric or Capistrano require management of the idempotency process within the tasks themselves (e.g you are responsible for making sure that running the same task again – regardless of success or failure – will not affect the server state). This is not an easy issue to overcome. 

Connectivity

Managing your server is only possible if a connection from the management server to the host is available. In the case of a connection failure, you are unable to control the state of your servers.

You Don’t Have Any Agents

Find it strange that this is both a pro and a con? We’ve covered many pros to an approach without agents, but that doesn’t mean they’re all bad. Agents do come with overhead, and utilization and maintenance issues that have some negative points, but they also have benefits, including monitoring, logging, and other functionality you may miss out without them.

Two often debated aspects of remote execution are security and agent functionality, though these can both be viewed as double-edged swords.

From a security standpoint, this is because while some claim security is compromised in remote execution due to SSH port exposure, others say that anything requiring installation brings a security tradeoff – you’re only as secure as the software you’re installing, and the access it has to your server.

When it comes to agents, let’s take Cloudify and Puppet as examples. Cloudify’s agent offers metrics, and Puppet brings reporting. Going the agentless route means you’re missing out on these features. Ultimately, it’s important to truly weigh the benefits vs. tradeoffs to decide what side of the coin best suits your environment and your needs.

How it Works

So, through some pros and cons, we’ve given a bit of insight on remote execution, and some ways to evaluate whether or not it’s suited for you and your environment. Now, let’s take a high level look at how it is generally used, and some of the operations you can perform with remote execution.

Fabric is a Python library and command-line tool that can be used to execute remote SSH commands on servers. Primarily, Fabric is used to deploy applications but can also be used for general-purpose server admin.

With Fabric, there is a layer of abstraction above managing server lists to execute commands on, to configure the SSH connection parameters (timeouts, retries, agent forwarding, users, passwords, disabling known hosts, SSH keys, and more), to configure server-specific roles, and more.

Here’s an example of a Fabric task:

<script src=”https://gist.github.com/shar1z/c252f690a625f797ad59.js“></script>

The above will provide a list of all servers defined under aforementioned roles, then execute (either in parallel or serially, depending upon the configuration provided) the above commands on them.

In this example, the servers for each role are static.

An example with a bit more complexity than Fabric’s default implementation, is a Python module called fabric-aws written by the folks at EverythingMe, which is an extension for Fabric. It enables you to configure auto-scaling groups (even from CloudFormation), specific instances, or instances based on filters for execution.  See the code snippet below for how we use this for deploying Hubot, our ChatOps bot.

Fabric also enables you to return the execution output so that it can be parsed and then (hopefully) analyzed. It also allows you to copy files to and from the server, and to generate files from templates to create files on the fly with different configurations when deploying to servers.

The Cloudify Fabric plugin allows remote execution commands on hosts without having to install the Cloudify agent on them.

To run commands with the plugin, there are a few options:

1. Write commands directly in your TOSCA blueprint
2. Provide a file with Fabric tasks, which you can then execute
3. Provide a Python module installed somewhere on the manager that contains the task and will then execute it

This approach is useful because the entire infrastructure can be managed using only the Cloudify Manager. This makes it an approachable option for companies with more strict policies for installing agents, as they can take advantage of the benefits of remote execution.