Bootstrapping Cloudify 4 on OpenStack, AWS, Azure, and more – A Simple Step by Step Tutorial

The most up-to-date tutorial on environment setup can be found at this link

Introduction

Before getting started with this tutorial, I would like to clarify something. This post will deal with a different, slightly more intensive process for getting started with Cloudify. In a preivous post, we walked users through installing Cloudify on OpenStack or AWS using a VM with a pre-baked Cloudify image, which meant that you got Cloudify with any plugins and add-ons we included. The below post gives users the bootstrap process, which means you will first bring up a clean virtual machine on your cloud and then install Cloudify on that machine. There are reasons to use the bootstrap method versus the image install method, but we won’t get into those here.
So, in this post, we will walk you through bootstrapping Cloudify Manager on OpenStack. This same process can be applied to other clouds as well, including AWS, Azure, and Google Cloud (GCP) – you simply need the inputs relevant to your IaaS.
To ask a question or report an issue, please visit the Cloudify users group.
NOTE: Keep in mind that this method of bootstrapping Cloudify Manager is not necessarily ideal, for various reasons, but it will give you the best understanding of what is required to get your environment set up with a manager and should work the very first time you try it (barring unforseen circumstances based on your environment or cloud setup).

Prerequisites

There are a few things to keep in mind and prepare before getting started, so please read through this post carefully at least once and then begin.

• Python 2.7 installed on your computer.
• Virtualenv (or similar) installed on your computer. Virtualenvwrapper is a suggested addition.

• IaaS Cloud provider and API credentials and sufficient permissions to provision network and compute resources (a new, clean environment is always best):
• AWS Credentials OR
• Openstack Credentials – skip steps 5 and 6 in those instructions – OR
• Azure Credentials

Learn more about cloud orchestration!  Go

Getting Started

It is recommended that users run Cloudify in a virtual environment such as virtualenv. Once you have created your environment in the terminal (or command prompt or Powershell), continue with the Cloudify CLI installation.

1. Install Cloudify CLI on your computer by downloading the binary file for your operating system (Linux, Ubuntu, or Windows) and install. If you are using a Mac, just run pip install cloudify in the terminal and it will install for you. You can run ‘cfy ——version’ to verify the install worked correctly.

2. Download and extract this blueprint archive to the directory (folder) of your choice and extract the contents, AND make sure your terminal is currently pointing to the directory to which the files were extracted.

Important Notes Before Step 3

A few important notes before we continue:

• Since this video is about bootstrapping a manager rather than installing a Cloudify Manager image, it is important to make sure you use a generic Centos 7 machine. You should be able to find a generic Centos 7 image in your OpenStack under Compute -> Images. If not, ask. your IT administrator.
• If the Identity (Keystone) API version on your OpenStack is set to “v3” (see below screenshot), you will need to download the OpenStack RC file to your computer, open it in a text editor, change the “export OS_AUTH_URL=https://xxxxxx.com/v3” to “export OS_AUTH_URL=https://xxxxxx.com/v2.0”. If your OpenStack Identity is already set to “v2.0”, ignore this and the rest of this list.
• Another step that can cause problems for users is ensuring the OpenStack configuration path is pointing to your RC file. So, just to be on the safe side, in your terminal enter export OPENSTACK_CONFIG_PATH=~/[PATH_TO_DIRECTORY_WHERE_RC_FILE_IS]/[RC_FILENAME]
• Then, go back into the terminal and enter source [PATH_TO_RC_FILE]/[NAME_OF_RC_FILE] in order to ensure your environment is correctly setup.

For AWS, you will need to open the blueprint file called “aws-blueprint.yaml” find the “cloudify_ami” section and follow the directions listed in the description. This includes changing the AMI, username, and block device in two places inside the blueprint.
For Azure, you will not need to change anything.
3. Install your environment’s infrastructure by executing one of the example commands below, inserting your account credentials where indicated. The simplest way to do this is to copy the text to a text editor, add the details, and the paste them into the terminal.
Note for AWS users: This process will automatically be run on US-EAST-1 (N. Virginia). For slightly more advanced users who may want to use a different region, open the “aws-blueprint.yaml” file in a text editor to customize the inputs.

For OpenStack run:

$ cfy install cloudify-environment-setup-latest/openstack-blueprint.yaml -i username=[INSERT_YOUR_OPENSTACK_USERNAME] -i password=[INSERT_YOUR_OPENSTACK_PASSWORD] -i tenant_name=[INSERT_YOUR_OPENSTACK_TENANT_NAME] -i auth_url=[INSERT_YOUR_OPENSTACK_V2.0AUTH_URL] -i region=[INSERT_YOUR_OPENSTACK_REGION] -i external_network_name=[INSERT_YOUR_OPENSTACK_EXTERNAL_NETWORK_NAME] -i cloudify_image_id=[INSERT_YOUR_OPENSTACK_CENTOS_IMAGE_ID] -i ubuntu_trusty_id_examples=[INSERT_YOUR_OPENSTACK_UBUNTU_TRUSTY_IMAGE_ID] -i small_openstack_image_flavor=[INSERT_YOUR_OPENSTACK_SMALL_IMAGE_FLAVOR_ID] -i large_openstack_image_flavor=[INSERT_YOUR_OPENSTACK_LARGE_IMAGE_FLAVOR_ID] --task-retries=30 --task-retry-interval=5 --install-plugins

For AWS run:

$ cfy install cloudify-environment-setup-latest/aws-blueprint.yaml -i aws_access_key_id=[INSERT_YOUR_AWS_ACCESS_KEY] -i aws_secret_access_key=[INSERT_YOUR_AWS_SECRET_ACCESS_KEY] --task-retries=30 --task-retry-interval=5 --install-plugins

For Azure run:

$ cfy install cloudify-environment-setup-latest/azure-blueprint.yaml -i subscription_id=[INSERT_YOUR_AZURE_SUBSCRIPTION_ID] -i tenant_id=[INSERT_YOUR_AZURE_TENANT_ID] -i client_id=[INSERT_YOUR_AZURE_CLIENT_ID] -i client_secret=[INSERT_YOUR_AZURE_CLIENT_SECRET] --task-retries=30 --task-retry-interval=5 --install-plugins

4. Get info to bootstrap and configure Cloudify Manager by running cfy deployments outputs in your terminal.
The output should look like this:

For the purpose of this tutorial, you will only need to follow the “Bootstrap” and “Configuration” steps. Ignore the “Demo” section. This will ready our environment to run the webapp we will deploy in a future post.

Bootstrapping

5. Bootstrap Cloudify Manager
It is easiest to copy and paste the outputs from the previous step into a text editor so you can easily copy and paste the info into the terminal. The below commands are for illustration purposes.

Upload an SSH key to the manager VM:

$ cat ~/.ssh/cfy-manager-key | ssh -i ~/.ssh/cfy-manager-key
 cfyuser@**.**.***.*** 'cat >> ~/.ssh/key.pem && chmod 600 ~/.ssh/key.pem'

_Note: Answer yes when prompted.

Install the Cloudify CLI on the manager host:

$ ssh -t -i ~/.ssh/cfy-manager-key
 cfyuser@**.**.***.*** 'sudo rpm -i
 http://repository.cloudifysource.org/cloudify/4.0.1/sp-release/cloudify-4.0.1~sp.el6.x86_64.rpm'

You will see output like this:

You're about to install Cloudify!
Thank you for installing Cloudify!

Execute bootstrap:

$ ssh -i ~/.ssh/cfy-manager-key
 cfyuser@**.**.***.*** 'cfy bootstrap
 --install-plugins
 /opt/cfy/cloudify-manager-blueprints/simple-manager-blueprint.yaml
 -i public_ip=**.**.***.*** -i private_ip=10.10.0.4 -i
 ssh_user=cfyuser -i ssh_key_filename=~/.ssh/key.pem
 -i agents_user=ubuntu -i ignore_bootstrap_validations=false
 -i admin_username=admin -i admin_password=admin'

Expect this to take 15-20 minutes.
When you see the following output, the manager is up:

Bootstrap complete
Manager is up at **.**.***.***
##################################################
Manager password is admin
##################################################

6. Configure your manager:
At this stage, it is suggested to wait 5 minutes for all of the services to synchronize.

Initialize the manager CLI profile:

You need to initialize a manager profile in order to control your manager. Copy the text from your outputs in the previous step and paste it in your terminal. It will look like this:

$ cfy profiles use -s cfyuser -k ~/.ssh/cfy-manager-key -u admin -p admin -t default_tenant **.**.***.***

Upload the plugins for your manager:

Note: the exact plugins you need to upload will vary. For this example, you will be shown the plugins to upload in your outputs.

$ cfy plugins upload
    http://repository.cloudifysource.org/cloudify/wagons/cloudify-azure-plugin/1.4.1/cloudify_azure_plugin-1.4.1-py27-none-linux_x86_64-centos-Core.wgn
Uploading plugin http://repository.cloudifysource.org/cloudify/wagons/cloudify-azure-plugin/1.4.1/cloudify_azure_plugin-1.4.1-py27-none-linux_x86_64-centos-Core.wgn...
Plugin uploaded. The plugin's id is 82568a34-f665-4677-af14-16575ea6c0c1
$ cfy plugins upload 
    http://repository.cloudifysource.org/cloudify/wagons/cloudify-diamond-plugin/1.3.5/cloudify_diamond_plugin-1.3.5-py27-none-linux_x86_64-centos-Core.wgn
Uploading plugin http://repository.cloudifysource.org/cloudify/wagons/cloudify-diamond-plugin/1.3.5/cloudify_diamond_plugin-1.3.5-py27-none-linux_x86_64-centos-Core.wgn...
Plugin uploaded. The plugin's id is 04efe149-ad8a-4ce1-b840-b0556a6efc18
$ cfy plugins upload
    http://repository.cloudifysource.org/cloudify/wagons/cloudify-diamond-plugin/1.3.5/cloudify_diamond_plugin-1.3.5-py27-none-linux_x86_64-Ubuntu-trusty.wgn
Uploading plugin http://repository.cloudifysource.org/cloudify/wagons/cloudify-diamond-plugin/1.3.5/cloudify_diamond_plugin-1.3.5-py27-none-linux_x86_64-Ubuntu-trusty.wgn...
Plugin uploaded. The plugin's id is adb4c1d5-d2b8-44b6-a4c2-3bef2a78a8f7

Create your secrets:

Adding secrets to your manager make your deployments more secure. The exact secrets you add also vary by clouds. Again, copy and paste from your previous step outputs and paste into your terminal. This sample below is based on Azure.

$ cfy secrets create -s ********* subscription_id &&
    cfy secrets create -s ********* tenant_id &&
    cfy secrets create -s ********* client_id &&
    cfy secrets create -s i********* client_secret &&
    cfy secrets create -s eastus location &&
    cfy secrets create -s pmrg2 mgr_resource_group_name &&
    cfy secrets create -s pmvn2 mgr_virtual_network_name &&
    cfy secrets create -s pms02 mgr_subnet_name &&
    cfy secrets create -s Canonical ubuntu_trusty_image_publisher &&
    cfy secrets create -s UbuntuServer ubuntu_trusty_image_offer &&
    cfy secrets create -s 14.04.4-LTS ubuntu_trusty_image_sku &&
    cfy secrets create -s 14.04.201604060 ubuntu_trusty_image_version &&
    cfy secrets create -s Standard_A0 small_image_size &&
    cfy secrets create -s 'ssh-rsa *********' agent_key_public
Secret `subscription_id` created
Secret `tenant_id` created
Secret `client_id` created
Secret `client_secret` created
Secret `location` created
Secret `mgr_resource_group_name` created
Secret `mgr_virtual_network_name` created
Secret `mgr_subnet_name` created
Secret `ubuntu_trusty_image_publisher` created
Secret `ubuntu_trusty_image_offer` created
Secret `ubuntu_trusty_image_sku` created
Secret `ubuntu_trusty_image_version` created
Secret `small_image_size` created
Secret `agent_key_public` created
$ cfy secrets create agent_key_private -s "$(<~/.ssh/cfy-agent-key)"
Secret `agent_key_private` created

Note that in the last command, the double-quotes are unescaped:
The deployment output was like this:

$ cfy secrets create agent_key_private -s "$(<~/.ssh/cfy-agent-key)"

But you will need to remove the on either side of the quotes so it looks like this:

$ cfy secrets create agent_key_private -s "$(<~/.ssh/cfy-agent-key)"

Your manager is now installed and configured!

6. When you are ready to uninstall your environment, run:

$ cfy profiles use local
$ cfy uninstall --allow-custom-parameters -p ignore_failure=true --task-retries=30 --task-retry-interval=5