Scaling Kubernetes Microservices on OpenStack With TOSCA Orchestration Pt I of II

In a previous post, I converted a Fabric-based plugin implementation to an Openstack agent-based implementation. In this episode, I finally reached one of the key goals of this long running effort; to automate scaling on Kubernetes using Cloudify native means. The path to this goal leads through some of the more exotic parts of Cloudify (the policy engine), a slightly unorthodox approach to metrics gathering, and makes use of another topic I mentioned awhile back, the deployment proxy.

Manage and orchestrate containers with Cloudify + Kubernetes. Try it.  Go

Overview

The latest iteration of the Kubernetes plugin adds support for orchetsrations that use the blueprint proxy, which permits a one-to-one correspondence between Cloudify blueprints and Kubernetes-hosted microservices. The latest version represents a culmination of one of the key goals of the project: hybrid cloud orchestration that support autoscaling Kubernetes. New features:

• a simple container hosting a Diamond collector to send metrics from the Kubernetes Pod to the Cloudify server.
• extended syntax in the plugin to support multiple Kubernetes descriptors per microservice. This to support Kubernetes service endpoints across multiple instances of the pod.
• additional logic in the plugin to recognize deployment proxy references, and treat them as though the foreign deployment is actually local.
• a scaling workflow for Kubernetes
• a Cloudify-native scaling algorithm for the embedded Riemann, that ultimately triggers the scaling workflow.
• a multi-blueprint approach using the deployment proxy. This results in separated, coordinated blueprints for Kubernetes itself, for the Kubernetes microservice pod, and the MongoDb instance.

The end result of these features is three orchestrations that work together; MongoDb running in the cloud (Openstack), Kubernetes deployed to Openstack; and The Nodecellar Nodejs app and a Diamond collector container deployed as a microservice pod (replication controller) into the Kubernetes instance. The Diamond collector delivers a count of connections to the Cloudify manager, which scales the number of pod instances in Kubernetes based on the connection count reported by the Diamond collector. This scenario demonstrates the scope of difficult orchestration problems Cloudify can handle.

Digging A Little Deeper

The Driving Vision

The changes in this iteration were driven by the desire to enable a specific hybrid cloud scenario:

1. Deploy, configure, and start Kubernetes on multiple nodes.
2. Deploy, configure, and start MongoDb on a cloud.
3. Deploy a microservice to Kubernetes.
4. Autoscale that microservice based on metrics received from the deployed microservice, by delegating to Kubernetes itself.

Using the Deployment Proxy

In order to separate the deployment of the microservice (Nodecellar), the deployment proxy was used. This permitted the overall orchestration to be composed of three blueprints. Participants in the deployment proxy pattern supply information to other participants via deployment outputs.

outputs:
  kubernetes_info:
    description: Kuberenetes master info
    value:
      url: {concat: ["http://",{ get_attribute: [ master_ip, floating_ip_address ]},":",{ get_property: [ master, master_port ]}]}

Here, the master node in the blueprint is advertised by exposing a URL consisting of its IP address and port. Inside the microservice blueprint, the master is referenced by a relationship to a proxy node:
The proxy:

  kubernetes_proxy:
    type: cloudify.nodes.DeploymentProxy
    properties:
      inherit_outputs:
        - 'kubernetes_info'

The reference:

  nodecellar:
    type: cloudify.kubernetes.Microservice
    relationships:
      - type: cloudify.kubernetes.relationships.connected_to_master
        target: kubernetes_proxy

The cloudify.kubernetes.relationships.connected_to_master relationship understands a both internal and proxy targets. The function of the relationship is simply to collect the IP and port of the Kubernetes master.

Enhancing the Microservice Syntax

Once the Microservice node has the information about the Kubernetes master, it needs to identify the Kubernetes descriptor files that the microservice consists of. In the case of the example blueprint, this consists of a ReplicationController descriptor (pod.yaml), and a Service descriptor (service2.yaml).

nodecellar:
    type: cloudify.kubernetes.Microservice
    properties:
        config_files:
        - file: pod.yaml
          overrides:
            - "['spec']['template']['spec']['containers'][0]['env'][1]['value'] = '@{mongo_proxy,mongo_info,ip}'"
            - "['spec']['template']['spec']['containers'][0]['env'][2]['value'] = '@{mongo_proxy,mongo_info,port}'"
            - "['spec']['template']['spec']['containers'][1]['env'][0]['value'] = '%{management_ip}'"
            - "['spec']['template']['spec']['containers'][1]['env'][2]['value'] = '%{deployment.id}'"
            - "['spec']['template']['spec']['containers'][1]['env'][3]['value'] = '%{node.id}'"
            - "['spec']['template']['spec']['containers'][1]['env'][4]['value'] = '%{instance.id}'"
        - file: service2.yaml

To support the needs of the Diamond collector, a special syntax was added to the overrides section parser. Tokens bracketed by %{ and } are evaluated by treating them as fields in the plugin context object. For example %{deployment.id} is evaluated as ctx.deployment.id during the plugin execution. A special identifier %{management_ip} causes the IP address of the management node to be substituted. These syntax conventions (along with the @{} syntax) are a little ugly, but serve the purpose. As before, The purpose of the overrides is to permit unchanged Kubernetes descriptors to be used in a blueprint.
The ReplicationController descriptor defines a Kubernetes pod that refers to containers with Node.js and the Nodecellar app, along with a container that runs a very simple Diamond collector that counts the number of connections to the Nodecellar port and forwards the metrics to the Cloudify server.

Collecting metrics

The actual collections of the metrics, pushing the metrics into the Cloudify server Rabbitmq instance, processing the metrics in a Riemann event stream, and triggering the scaling of Kubernetes, is left to my next post.

Conclusion

Creating a hybrid orchestration that demonstrates a full range of Cloudify capabilities has been the goal of the Kubernetes plugin effort. The effort has culminated in a plugin and blueprint that demonstrates full lifecycle management and coordination of Kubernetes, Kubernetes microservices, and Cloud services, including arbitrary metric triggering of automatic scaling behavior. The next post will dig into the metric gathering and scaling mechanisms. The code is on github. As always, comments are welcome.