DICE deployment tools – Cloudify Orchestration with TOSCA

This article was originally published on the DICE blog. Matej is Project Manager at XLAB. The article is being re-published with the author’s consent.
Introduction by Jeremy Hess

DICE is a European research consortium that is working on “delivering methods and tools that will help satisfying quality requirements in data-intensive applications by iterative enhancement of their architecture design.” In their research, they have found that Cloudify open source orchestration with TOSCA, is a great framework for use with their deployment tool. In our communication, Matej noted that his company, XLAB, is working “to give the developers a simple way to push their application to be deployed in a testbed or in a cloud.” We were very pleased to learn of XLAB’s interest in Cloudify for the DICE deployment tool, so we wanted to bring you some of their blog posts highlighting their very interesting work with Cloudify.

Cloudify – Intelligent Cloud Orchestration Done Better. Download Free.  Go

Let us suppose that you or your customer have a lot of data: collected in a database somewhere, or filling up from the daily activities. Or perhaps a set of IoT sensors costantly stream their measurements. You know about Map/Reduce or other modern ways of processing data. So now you need to design and create an application, which uses Big Data technologies, to mine the data and get real-time insights into the deeper aspects of your data. DICE will help you to quickly get from a concept to a fully deployed code in the test bed – when you start and whenever you change something.

The DICE tools aim at simplifying and speeding up the process of developing data-intensive applications. By this we mean that if you want to create an application, which deals with Big Data, you should not have to graduate from all the ins and outs of designing, testing and deploying your application. One of the bigger hurdles in using Big Data technologies is the process of installing, configuring and starting up the various services before an application can even start processing any data.

One of the great things about living in a modern world is that there are a lot of tools already available that solve problems of applying DevOps, treating infrastructure as code, and packaging complex installation processes into easily usable cookbooks. So potentially it should be an easy matter to, say, quickly whip together a few nodes that run a Zookeeper service, some Storm nodes and a Kafka messaging bus. Yet in our experience, the user always has to understand a lot of seemingly arcane details about each service, experiment with the community-provided cookbooks and tune configuration before the combination of services can start to work together.
In the DICE deployment tool, we bring together the good from various communities. First, we adopted an emerging industry standard for describing an application as a topology or a blueprint. OASIS TOSCA offers a simple, clear and efficient way of expressing the building blocks of any Cloud application: nodes, services, their relationships and any set of the needed parameters.

Next, a cloud orchestrator needs to consume this blueprint and execute the necessary steps to bring the blueprint to reality. In DICE, we use the Cloudify, an open source cloud orchestration framework provided by the GigaSpaces Technologies. Cloudify packs support for a wide range of public and private cloud platforms, existing blueprint constructs for expressing the required components of an application, and is also reasonably close to the TOSCA standard.

Cloudify serves as a mediator between an abstract description of an application on one side, and the tools that perform the installing and configuring on the other side. At this low level, one can use anything from simple bash and Python scripts to employing Chef cookbooks, Puppet recipes or any other configuration manager. Orchestration here means that Cloudify brings a notion of a whole application, so each individual node’s configuration comes with a context of the other interdependent nodes.

With the DICE deployment tool, we work to improve that by providing ready-made packages of libraries, which employ Chef cookbooks tested to work together. If you use the DICE deployment tool as a stand-alone component, then defining an application to be deployed is a matter of writing down the services by type in a simple YAML file, and the tool will do the rest.