Simplifying Hybrid Cloud Deployments with AWS EKS and Outpost

February 24, 2021
Posted by: Nati Shalom
Category: Hybrid Cloud

Challenges and failures in handling multi-cloud and hybrid projects (a.k.a digital transformation projects) are no secret – in fact I have been covering this extensively over the past year, most recently (and perhaps most loudly) in my recent Forbes contribution (Our Gritty Digital Transformation Reality — And How To Make It Better) where I pointed out the main reasons that lead to the complexity (and therefore failure) of hybrid cloud projects:

“Businesses are defining cloud opportunities narrowly with siloed business initiatives, and this practice practically guarantees failure. IT departments are unable to construct the architecture necessary for public cloud services, and the unhealthy assumption that “lift and shift” could be a shortcut to cloud transformation often results in the worst of all worlds. Taking real value from the cloud requires transformation to cloud-native architecture best suited to take advantage of cloud value.”

In this post, I aim to share lessons learnt from a recent joint project with AWS which involved a hybrid cloud strategy using AWS EKS as a Kubernetes cluster across the public and on-prem environment, AWS Outpost as the on-prem data-center and Cloudify as the service-orchestration that provided consistent end-to-end automation across both AWS public and on-prem environment (as well as the existing VMware-based stack).

Even though the project itself focused on a 5G use case, many of the lessons from the project can indeed be applied to any hybrid cloud use case.

How do AWS EKS and Outpost simplify hybrid cloud complexity?

In a nutshell- it provides consistent infrastructure and application workload management across these two environments.

Outpost provides consistent infrastructure and Kubernetes (EKS) provides consistent workload management. This consistency significantly reduces the complexity of running hybrid cloud environments between the public cloud and the corporate environment. Kubernetes also provides means to reduce the degree of lock in at the application workload level.

Consistent Multi Cloud Management with AWS Outpost and EKS

Adding consistent management between AWS and existing VMware/ OpenStack infrastructure.

AWS Outpost and EKS provide a great multi-cloud solution that is very AWS centric. It doesn’t address your existing infrastructure which is often based on VMware or OpenStack.

This is where a multi-cloud service orchestration such as Cloudify can help to bridge this gap.

Cloudify provides consistent management across public, private, and brownfield environments and allows users to manage VMware or OpenStack environments from within AWS.

Cloudify Consistent Multi Cloud Management between AWS and your existing VMware or OpenStack infrastructure.

Using AWS Lambda and AWS CloudFormation as a generic service wrapper

Multi cloud orchestration often involves the mapping of each cloud resource into template format. Each cloud provides its own proprietary templating automation language, AWS with CloudFormation, Azure with ARM. Terraform has gained popularity as it provides consistent infrastructure orchestration across multiple cloud environments.

Cloudify supports both AWS CloudFormation, Azure ARM and Terraform (as well as its own infrastructure mapping) in a cloud agnostic way to map different infrastructures (which is now supported by AWS), allowing users to utilize the right template for the job. You can find more details about this here.

Still, interacting with each cloud resource at the API level could be too complex and would make the automation template fairly verbose and hard to maintain. We found that in many cases wrapping the resources with Function could provide a fairly simple API wrapper which includes also custom logic that can reduce the verbosity level significantly as it can abstract the underlying resources and API to expose the specific interface that is needed.

Functions are also easier to maintain across the cloud, using a multi-cloud serverless framework to help avoid lock in – as described in this post.

Future work: Multi-Cloud at edge scale

According to this CNCF SURVEY, 80% of the organizations are using more than one Kubernetes cluster. The main use cases for having multi Kubernetes clusters include separating between public vs private cloud, separating Kubernetes application and teams and separating local development environment.

Edge computing is also becoming a popular use case for using multi Kubernetes clusters. Edge can be seen as a remote extension of the cloud infrastructure towards a specific geographical location – often used to provide more responsive application services

As we reduce the complexity of managing hybrid cloud environments through the likes of Outpost and EKS – this creates the opportunity to manage many sites that fit into this architecture.

Managing workloads across many sites brings with it its own set of challenges, resulting from the fact that we’re now managing many relationship scenarios between a service and the sites on which it runs. For example, a service can run on more than one site- How do we manage the deployment of this operation? How do we handle a partial failure scenario? How do we ensure that the binary needed to run the service is co-located. In addition to that were also we need to handle the security aspects as we ll as scalability challenges of such a distributed environment.

The upcoming Cloudify 6 comes with a specific solution for simplifying multi-site management across clouds and edge environments. This includes:

Single deployment to many sites
Built-in support for blue-green deployment
Managing 100’s of Sites and services
Separate sites (Environments) and workload deployments
Site Auto Discovery
Zero touch provisioning and Discovery

Cloudify 6 – multi-cloud and edge orchestration at scale

Many enterprises still find it hard to achieve synergy between their hybrid and multi-cloud environments as noted in the 2020 IDG® report:

“55% of organizations use two or more public clouds, but 79% struggle to achieve synergy across their multiple platforms.”

This AWS project allowed us to achieve a consistent hybrid and multi-cloud environment using a combination of AWS Outpost, EKS, and Cloudify, thus removing the complexity of such environments, allowing us to handle even more complex use cases involving many sites where edge marks the far end of multi-site use cases.

Handling such a distributed environment opens a new set of challenges that we now need to address including deployments of the same service across multiple sites as well as security and scalability challenges. This will become the focus of our next release, so stay tuned!

Nati Shalom, CTO, Cloudify

References

Take Cloudify for a Test Drive

Start your free trial Visit Our Documentation Join Our Slack Channel