Introducing Cloudify Insights – Multi-Cloud Cost and Usage Analytics In A Single Dashboard

Introduction

In our last blog post, we talked about the new and improved features in Cloudify UI and how to utilize them to manage and control your multi-cloud environments. In a previous blog post, we took a look at the next generation of our Cloudify Composer tool which allows anyone to create TOSCA blueprints in a simple drag-and-drop manner, without the need to do any coding. We are continuing our tour of the Cloudify 4.2 release goodness with a new feature.

Watch our upcoming Cloudify 4.2 webinar to see what the new version is capable of!   REGISTER NOW

Cloudify Insights

Without further ado, we are really proud to introduce Insights – a brand-spanking-new feature that is sure to make IT managers happy campers. Our engineering team, along with the great team at Mist.io, has developed a widget for Cloudify UI that presents analytics on cloud costs and usage across all of your environments. The goal of this tool is to allow IT managers to control and reduce usage and costs for their hybrid and multi-cloud infrastructure.
As an add-on to the Cloudify UI, Insights provides end-to-end environment monitoring, and enables you to make the most informed decisions regarding your environments, taking into consideration everything that matters.

Insights Reporting

The Insights widget reporting shows the following information:
    1. Cost Overview – Real-time and historical VM cost reporting
    2. Utilization Overview – Real-time and historical VM usage reporting
    3. Machines Overview – VM provisioning reporting
The filters in the widget enable you to drill-down as far as a single deployment to see individual cost and usage. Metrics can be presented by time period (hour, day, week, month, quarter), cloud (IaaS), and specific Cloudify deployment.
Insights also provides a comparison option for different timeframes. For example, it can show that this month’s Azure costs were lower than last month’s, or the AWS costs were higher than GCP this month for the same deployment.
Let’s go just a bit deeper into the three reports.

Cost Overview Report

This report provides real-time and historical cost reporting for VMs. You can see time period cost comparison, the VM costs by cloud, and VM inventory per cloud or per deployment.

Usage and cost trends over time and by cloud

Utilization Overview Report

This report presents the amount of total/min/max/avg VM cores and loads, across all clouds. It can show time period load comparison, and also total cores and load per cloud.

View of VM usage efficiency

Machine Overview Report

This report provides the real-time and historical VM provisioning activity of max/avg/total, or per cloud. It also presents the associated costs, and can provide a comparison of VMs provisioned per given time period.

VM provisioning activity over a specific time period

Supported Clouds

Insights was built to support a very large number of cloud platforms.
Public

• AWS EC2
• Azure Classic
• Azure Resource Manager
• Google Compute Engine
• IBM (SoftLayer)
• Rackspace
• Digital Ocean
• HostVirtual
• Linode

Private

• OpenStack
• VMware vSphere
• VMware vCloud
• KVM
• Packet.net
• Vultr

Installation

To use Cloudify Insights as a service, or on-premises version, please contact us as this requires an additional subscription model. You can, however, evaluate the service by following the steps below:

• The service’s installation is an add-on to a running Cloudify manager.
• Before following the installation steps below, make sure you have a manager up and running and a Cloudify CLI connected to it.

Insights Manager Agent Install Procedure

Insights installation includes 2 phases – the first will be the installation of an AMQP Middleware allowing the service to access the Cloudify backend, and the second will be the import of the widget to the Cloudify UI.

Installing the Insights AMQP Middleware

    1. Download and unzip this file.
    2. Edit the YAML file which includes the inputs for the deployment of the Insights service:
Open the archive, and edit the file local-blueprint-inputs.yaml which is located in the amqp-middleware-blueprints folder. The inputs file comes with several configuration options, most of them, however, already have default values. All configuration options come with detailed descriptions regarding their purpose.
The inputs file can be broken down into the following main sections:

• Cloudify Manager Inputs: These are the inputs required for the AMQP Middleware to be able to connect to Cloudify Manager’s RabbitMQ server, as well as query the Cloudify Manager REST API in order to retrieve information on deployments and node instances.
• User Inputs: This inputs section requires user information in order to create a new Insights SaaS account.
• Cloud Credentials: Users may specify their cloud credentials and the AMQP Middleware will take care of adding the user’s infrastructure to their Insights account.

Once all required inputs have been provided, save the inputs file.
    3. Use the CLI connected to the manager and execute the blueprint installation: cfy install <PATH-TO>/amqp-middleware-blueprints/local-blueprint.yaml -i <PATH-TO>/amqp-middleware-blueprints/inputs/local-blueprint-inputs.yaml
    4. Then execute cfy deployment outputs amqp-middleware-blueprints
You will now be presented with two outputs of the process:

• “Credentials” holds your admin Insights account token
• “Insights” holds the path to the Insights web app with a read-only token.

Keep the read-only token as you will need it for the widget configuration.

Insights widget configuration

    1. Download the Insights widget zip package here.
    2. Login to the Cloudify Manager as admin role and switch to edit mode.
    3. Add a new page and name it what you like (“Insight” is a good choice).
    4. Click on the page and then choose the Add Widget option.
    5. Choose the zip file you downloaded and click OK.
    6. Now, select the Insights widget and choose Add Selected Widgets. The widget will then appear on your page, but will not yet show data as it still needs to be configured and linked to your Insights account.
    7. Copy the read-only token which you got as an output of the Insights installation in step 1d, click the widget configuration icon, and paste the read-only token in the right configuration field.
    8. Click OK and in a few seconds, your widget should start showing the data collected in your Insights account.
Cloudify Insights can also be installed as an on-premise service. In order to do that, please contact us.

Adding Cloud Accounts

In order to add more clouds to an existing account, populate the inputs/update-clouds.yaml file. Its structure is the same as the corresponding clouds section in inputs/local-blueprint-inputs.yaml.
Then run cfy local execute -w add_cloud -p inputs/update-clouds.yaml

Removing Existing Cloud Accounts

To remove an existing account, edit the inputs/remove-clouds.yaml in order to remove unwanted clouds. Remember to refer to each cloud by its title.
Then run cfy local execute -w remove_cloud -p inputs/remove-clouds.yaml

Manager Restore

In case you wish to make a snapshot and restore your manager, you will have to perform a teardown of the service, by running the uninstall workflow on the Insights deployment, then delete both the deployment and the blueprint. You will then have to perform the snapshot creation and restore process on the new manager, and reinstall the service again on the new manager.
For more information, visit the Github repo for this project.