Welcome to the very first episode of Cloudify Tech Talk where we take a deeper dive into Terraform and how it fits into Cloudify. CLICK HERE for Cloudify & Terraform integration in action- supporting material for this podcast.
Episode One – All about Terraform
[00:01] Jonny: Hi, everyone, and welcome to the very first episode of the Cloudify Tech Talk podcast. My name is Jonny, I’m with Cloudify. I’m going to be your co-host today. Today’s session will focus on Terraform. Everything about Terraform, and in particular, how it relates to Cloudify. Without further ado, I’m going to hand over now to my co-host Ilan Adler, who will give you more information about our speakers and the format of today.
[00:26] Ilan: Thank you, Jonny. So to get started, I’ll introduce some of my colleagues here. So we have a Cloudify founder and CEO, Nati Shalom. We have Alex [Inaudible 00:37], who’s the Cloudify R&D director, as well as Isaac, one of our senior engineers from the Cloudify field team. Nati, usually Terraform, the biggest or should I say the million dollar question that comes up with Terraform as I get it from customers, and users all over is what’s different from what’s Cloudify doing to Terraform? How exactly—how does Terraform do this and Cloudify do that? And why do I need both? Or why do I need one or the other? So I’d be happy for our users and our audience if we can start off with addressing this general question.
[01:24] Nati: So this is an excellent question. And probably the most frequent getting on set in a couple of minutes, even though it’s probably worth, fear, discussion, terminology. When the couple of years ago when orchestration come to the world, everything that has to do with automation was considered competitive. So we had those questions of is Terraform compete with Kubernetes, is Kubernetes compete with Ansible, is Ansible compete with Terraform. And obviously Cloudify was part of those discussions as well. And a couple of years today into this, what we’re starting to see is that there’s a greater—I would say realization that there is no one orchestration tool at all. And we’ll talk a little bit about what that means and later through this discussion.
We’re starting to see each of the orchestration kind of taking its own special domain and focusing on that. So there’s kind of virtualization of that model; Terraform being the infrastructures code kind of a specialist. Ansible, the configuration management. Kubernetes is the container management platform. And the question was, well Cloudify [Inaudible 02:35] into the…into all that and we’ll talk more about it in the discussion. But what we’re seeing is that Cloudify is pretty good at handling heterogeneous environment and handling distorted environments and also managing other orchestrator, being the multi domain orchestrator as it’s called. And that’s kind of where we kind of see the synergy between them. And again, we’ll talk a little bit about those use cases as we go along, and I think that will come very clear.
[03:03] Ilan: Excellent. So before we sort of explain more on the growing demand, Alex, who’s recently joined Cloudify, has some insights that he had regarding his past work with Terraform and some of his experience using Terraform and I’d be happy Alex, if you can start us off with some more of that experience and your point of view.
[03:34] Alex: Yes, sure. So I’ve been with Cloudify for one month already in but my previous company was a sass company, a startup. We use the—heavily using Terraform, we use Kubernetes. And the way we use the Terraform it’s very interesting. So for each developer…each developer had his own environment and the way to provision them they actually—we’re using Terraform templates. And in order to provision those environments, we actually had to run it manually in for QA testing environments that were on demand. So it was a tightly bonded into our [Inaudible 04:20]. But going to production again, we wanted to make it more controlled.
So actually, we wanted to make sure that all the templates or doing whatever we desired to do. In the agreement was that the product the production will be a manual. Before we had different challenges because each developer provision on environment and the environments are pretty similar and we did manage to face kinds of challenges when a new developer had to introduce in new infrastructure resource, and then it’ll actually update some repository that everyone using in and of course, once you provisioning in, you’d never environment without Terraform in. You’re facing all kinds of edge cases. But Terraform by itself sold us lots of lots of automation around infrastructure and all this area.
And once I finished actually, once I joined the Cloudify and first when I heard about Cloudify, the notion of bringing together Terraform and other orchestrations that provisioning actually application of construction. Well, it totally makes sense in for me, at least because now we know for sure that it’s not enough. For example, if the previous company was having, we’re using Kubernetes so it’s not just enough to allow photos and rely that everything works. It has to be bonded together with the [Inaudible 06:08] infrastructure. So instead of fighting all the time to have each one’s kinetics and Terraform updates, it’s really nice when you have one place that makes it makes sure that everything is aligned up to date. That’s my thoughts from my experience.
[06:32] Ilan: Excellent. So that’s a great, that’s great info on how sort of Cloudify plays into that role. And that’s I think a fantastic segue for us to introduce Isaac. Like I said, one of our senior engineers and Isaac works with our customers and he knows he can describe some of the growing demand or how he’s seen customers, either using or thinking of using Cloudify with Terraform.
[07:02] Isaac: Okay, so what we’re seeing is a trend to—instead of trying to fit the problems to one particular tool, we see a desire to fit tools to problems depending on circumstances. So it could be basically a different choice for tooling depending on application teams and infrastructure of choice. For example, if I want to spin off database service I’ll probably want to use cloud formation or [Inaudible 07:38] arm. If I want to spin VMs across clouds, I will probably use Terraform. For configuration, I’ll use Ansible and for container management I can use Kubernetes and inside Kubernetes if I have a private cloud, I’ll probably want to use, you know like something like in open shift. For public cloud, I’ll use GKE, EKS, APS, and so forth. So what we see is, there’s a demand for the ability to control all of these tools across all of these platforms, rather than be forced to use a single tool for all of the problem domains that are involved.
[08:20] Ilan: Right and I think that’s an excellent, excellent point and Nati can address that more but the idea of forcing different especially in larger organizations with multiple development teams, and each one likes to use their own tools, and it’s not just what’s the best tool but also tools that they’re—they have the right skill sets and they’re used to using them. And then force a tool to replace them or force one tool, like you said to solve all the problems. Organizations are finding out that that doesn’t really work and they need to figure out how they can [Inaudible/Cross talking 08:58]
[09:00] Isaac: Yeah, the price for this is just too high. Try to fit the problem to the tool.
[09:05] Ilan: Exactly and that’s like I said, a fantastic way of looking things. And as—in addition to that, Nati I’ll sort of shoot this to you. Can you explain how some of the things that Alex and Isaac mentioned how they came into the Cloudify choice to integrate with Terraform and again, more of some of the key differences between the two platforms?
[09:29] Nati: Yeah. So I think by the time of this question, I think it’s clear from Alex and Isaac’s description that everyone is realizing there’s no one tool to rule them all. Indeed, when we started Cloudify back, I think in 2014, we were thinking about, you know Tosca, and creating one language that will fit to every cloud, to every infrastructure, to every tool. That was the initial vision. I think the reality shows that if you go in [Inaudible/Distorted Audio 09:59] there’s no option to build such a language or tool. And none of the tools can actually cover such a widespread problem. And that kind of forced us to ask ourselves that question of what is our core? Assuming that Terraform is pretty good at doing infrastructure as code.
Ansible, as Isaac mentioned, is doing configuration management and Kubernetes does the container management platform, what is the space that Cloudify needs to cover and we looked into that. What we found is that Cloudify is pretty good at managing heterogeneous stack in managing other services and not just infrastructure elements, especially when we introduce a new feature in the kind of the latest release, which is called service composition, meaning the ability for us to manage other orchestration as if they were just another element in the Cloudify templates. What the service orchestration—what the service composition feature really allowed us is to create this nested type of blueprints that can be compostable from many other type of blueprints, not just the Cloudify blueprints.
It could be composed from a Terraform template, it could be composed from a Kubernetes template, it could be composed from [Inaudible 11:16]. And I can create that mesh of services as one end-to-end automation, but not necessarily in one language. And that really opened us to think about this as the main problem. How do we use Cloudify as the multi domain orchestration, as the thing that connects all the things together to allow users to at the end of the day, get an end-to-end automation and end-to-end view of the entire workflow across those different domains? And that was an interesting challenge for us because clearly, if we want users to use that feature, we need to make sure that for Cloudify users, the integration with the likes of Terraform would look native as if you were running just another blueprint from Cloudify.
So how do we make that experience in the UI, in the topology view, in the composer, all the features that Cloudify have, the workflow, all those things and the logging. All those things needs to look native for Cloudify users. Similarly, when we look at Terraform users, we don’t want to now force them to use Cloudify to use Terraform. And now force them to run everything through Cloudify, just use Terraform. Because some of the Terraform user would just like to use Terraform the way they used to. And so how do we synergize the two? How do we really bring the Terraform experience and the artifacts that are the output of Terraform into Cloudify but still allow those users to maintain…that kind of exposed a very interesting challenge for us and that’s kind of the goal that we set ourselves in terms of the integration itself.
And, the other thing that we added on top of that is really the ability to allow users not just to endow to use Terraform as a native tool within Cloudify, but also allow users to have a selection of choices of infrastructure. So they could use [Inaudible 13:02] arm, or they could use Terraform or use cloud formation r they can use Cloudiy pretty much in the same way and decouple that question from the application users. So the application users can choose almost at any point in time, which infrastructure choice he is going to use, which infrastructure tool he is going to use to provision his environment.
In many cases, that choice would be just because there is already something out there, someone already wrote a cloud formation template, especially if it’s in the case of Amazon to provision RDS as Isaac mentioned. Someone already created a template to create my network security aspects and configuration. So a lot of those choices will probably come from that reality rather than just a selection because of the flavor, our experience, things that I think will happen, and that kind of open other things, but I’ll leave that for the next iteration. So hopefully that’s a long answer to a short question, but hopefully I answered the question. Ilan back to you!
[14:05] Ilan: Nati you mentioned some of the things so if you can quickly take us through some of the main features behind the integration and sort of the things that we did just in bullet points can also be helpful. And obviously, when we post this we’ll also refer to relevant information.
[14:25] Isaac: Our plugin provides the functionality first and foremost to orchestrate Terraform itself, if you don’t have the Terraform installed; we will install it for you. We will download plugins and put them in the plugins directory and so forth. And when it comes to Terraform templates, we offer the functionality of applying a template, destroying a template and also we enabled two way, sorry, we enable a bi-directional sync of state between Cloudify and Terraform. So you could, for example, orchestrate a Terraform module, which will create [Inaudible 15:17] cloud. If resources change on the cloud, you can import these changes back into Cloudify.
And the other way is you could update your cloud by updating Cloudify no template with an updated Terraform template. So that provides two way synchronization between Cloudify state and the state in the cloud through Terraform. So the way that we offer passing parameters is you could—when you orchestrate Terraform module using the plugin you provide variables to the templates and you provide them through parameters in the node template itself. So you can grab values for Terraform variables from blueprint inputs to be from attributes that belong to other node instances.
We simply take these values and we pass them off to the Terraform template, the end of the orchestration of the module. And after every reload or a refresh of the template, we update the entire….so basically, what we do we take the entire Terraform state in the form of JSON, and we put it as a runtime property on the node instance. So you can then access the Terraform state from any other node template, or node instance in your blueprint or even in other blueprints.
[17:07] Nati: And in terms of the workflow that are currently supported. So there, you mentioned that there is a way to do bi directional sync to this. I think there’s also a way to check state and stuff around that. Can you elaborate on that?
[17:26] Isaac: So we offer a way to synchronize the state in the cloud by having Cloudify reload a Terraform template from a different source. So let’s say that you first orchestrated a module that will create all sorts of resources on the cloud. You want to update these resources so you could instruct Cloudify to reload the template from a different source that will be containing a different module, and there will be a call to Terraform to calculate the differences and apply these differences on the cloud.
That’s one way to synchronize, basically, from Cloudify to the cloud. The other way works by if, you know, your state changed on the cloud itself can pull these changes. And you do that by running a workflow on Cloudify, and what that workflow will do. It will do a state pool; it will call Terraform to do a state pool, to pull the updated state and then we take that updated state and we set it as a random property on the note template.
[18:48] Nati: And maybe one last question on that regard is the clustering aspects. How does that fit into the Cloudify clustering and availability model?
[18:58] Isaac: So, in Terraform you can store your state in a few ways and that’s called the back end, you can store the back end on S3, on basically many, many other mechanisms. And when you do that you are subject to that state being synchronized by that particular facility. For example, in S3, you would subject yourself to the availability and clustering and failover of S3. But when you use a file back end, which is a default in Terraform. What we do—we take that file and we encode it as basis 64 and we store it also as a runtime property in Cloudify. What that result is basically that the content of that state file will be available in the Cloudify database, and thus will be replicated to all other Cloudify cluster members.
So in other words, when you use a file back end in Terraform, which is the default, high availability of the state is going to [Inaudible 20:20] you by virtue of clarify replicating its own database. When you use a different back end, then of course, you’re subject to that other facilities availability.
[20:33] Nati: Excellent. And you’re also, I think Isaac, worked on an example that shows the Terraform integration, but also not just Terraform integration. Maybe you could say a few words about that new example and again, for audience that are listening to this, we’ll put reference to all those pointers that the notes section. So what does the example actually show?
[20:58] Isaac: So what the example is created? What does it—first creates a topology on AWS, creates a virtual private cloud and elastic IP a virtual machine, subnets security group isn’t just a small Virtual Private Cloud with a VM with a bunch of related cloud resources. And what that demo does is like after installing it, we change something in the PC to console, for example, we change the VM name, and then we run the workflow to refresh that Terraform state. And what it does, as I said before, it will take the state from the cloud and update the Terraform state, and then update the cloud fire one time property that represents that state. So we also show how that information, the updated information is available to you by simply looking at deployment capabilities which are calculated dynamically.
So that’s how we show one way of the synchronization. And we show the other way of synchronization by executing the reload function by pointing clarify to an alternative source for the module. That alternative contains another slight change to the cloud topology. So we run the reload and we see how the cloud configuration like sorry, the state of the resources on the cloud change, and also pulled back into the Terraform state, and again into the Cloudify state to be available for other consumers. So we show both ways of the synchronization. And of course on top of that, we show them the orchestration of Terraform itself, the installation of Terraform and installation of plugins at the beginning and the deletion of plugins and Terraform itself at the end
[23:04] Nati: Excellent: And then for those who wouldn’t have time to look into that there’s also a way for you to run the application, in this case, node JS or another application and kind of see how that works also in a consistent way with other orchestration, like Azure Arm or cloud formation, which I think touches on some of the points that I mentioned, in terms of experience. Alex, what’s coming next in terms of our plans to integrate Terraform into Cloudify in a more native way?
[23:35] Alex: So regarding Terraform, in the coming to the release, we are planning to actually bring the topology that is created by turf and tested inside the controller. Not that you’ll be able to manage templates to composers but the infra, the basic tools that you asked the other intelligences that were provisioned by Terraform, they’ll be visible inside the component, the deployment itself, actually. So currently a derivative trick is you can see only that Terraform template was executed, not presenting what exactly was provisioned. And now we’re actually extending it. So there’ll be a better visibility on what actually happened in the template.
[24:28] Nati: Excellent and we’ll also be able to see on the topology view a little bit more insights into the actual underlying topology of Terraform and see the state of the deployments or the resources being provisioned, right?
[24:45] Alex: Yes, correct. So everything was provisioned or going to be actually changed. So that would be visible inside the corner inside that topology.
[25:01] Nati: Excellent. So I think Ilan this kind of brings us to probably the last question on this discussion. And I’ll let you continue.
[25:13] Ilan: What sort of users, organizations should actually be looking to use this sort of integration with Terraform, with Cloudify? Is it both existing Cloudify users, is it people who are using Terraform and want to extend, if you can just enable a little more clarity into that could be a good summary of what we’ve discussed.
[25:40] Nati: Actually want to point to Alex because Alex has been using Terraform in his previous company and kind of Alex in kind of retrospect, if he would have Cloudify that a build there. Would you use it? In what context would you use it, if you would use it and obviously, if you can, in terms of what you think would be the general use of it?
[26:04] Alex: Sure. So as explained previously, I can see and that’s lovely along with the idea of yes, that infrastructure and application layer that are bonded together as long enough to separate and in big groups that working on the same project, it’s very important to put—to align the infrastructure with application layer. Now, if you’re looking back and retrospective how I managed to actually align both Terraform infrastructure and application layer, for example, still for development environments, environments that actually provision manually. It’s still a big gap and it’s very important to make sure that environment is on the right—is aligned.
So it’s much easier when you have actually one point that triggers in and make sure that infrastructure and application than the same version and what application especially for infrastructure, it’s actually there and not try to play with the versions and deploy each separately. So actually to keep this consistent, not only in the NSACI, but on the manual environment to [Inaudible 27:34]
[27:35] Nati: So maybe I’ll just, again, my perspective on that, and I think it’s just a continuation to what Alex said is that what we’re seeing, especially in the context of CICD type of context is that many organizations are now tasked, especially the infrastructure group within the organization, have been tasked to provide this consistent way to access the infrastructure and what they found out is that each group took their own decision of tools, but also their own way of using those tools, and there’s lack of consistency around that.
And so they found that the group that are now being tasked to put some order in that system, and what that group is doing is putting a lot of integration work between the tools and trying to provide common infrastructure to serve their different application groups. What I think they will find, like in any other infrastructure project, is that this is not just a bunch of scripts that a group can be writing. There is a—it’s getting very quickly into a sort of project. And this is where the build versus buy question becomes relevant. And what we’ve seen in one of the recent Gartner report also is a clear—something that came out of a survey that they’ve done last year between I think 300 DevOps users around that question.
That came clear is also realization that organization and are finding themselves with many silos and many tools and they’re getting lost in how to manage it. So they’re looking for a platform that will allow them to manage all of them, especially in a CICD kind of pipeline. So what does that mean in terms of the CIDC pipeline? I’m going to have not just one user that’s going to use all those three tools. I’m going to have different users; maybe each user would use one of those tools, but want to serve them through a consistent interface, for governance, for control for other things. And that’s kind of where the concept of what we call environment as a service comes handy.
And we see Terraform integration, being part of that concept that we’re introducing in which as Alex mentioned, we’re allowing users to decouple the elements of infrastructure from the application. And in that case, allow the infrastructure group of the integral organization to serve the different demands from the different organization and different groups that they need to serve within organization in an agile way, but still in a controlled way, in a consistent way. So that’s kind of where we started to see it. So it’s really, really targeted for those larger organization that needs to provide common infrastructure across the different groups and needs to have some order in the way they step up tools are being used.
And that’s kind of how we looked into that. And so when we do the integration with Terraform, it’s not just for the sake of integration with Terraform. We’re looking at from a bigger picture of integrating all the rest of the tools of the organization and I mentioned that already during this call.
[30:30] Ilan: Absolutely Nati and actually one of the biggest learning points that I have in conversations with users in regards to what you say is the most—the hardest thing to do at the end of the day is usually not technical. It’s mostly a cultural slash structural and forcing people like Isaac said into a tool because we have this tool is very, very difficult and usually doesn’t work and obviously is not cost efficient and all that. And that’s I think one of the biggest things that organizations are learning and this is a good summary of the points that you guys mentioned, and how this integration makes sense on a larger scale. With that, I think we’ll commence. I’ll hand it back over to my co-host.
[31:25] Jonny: Thanks Ilan -so in two weeks we’ll be back. We’re going to be taking a deeper dive into Kubernetes. And yeah, I want to thank everyone for participating thanks to Nati, thanks to Isaac, thanks to Alex. Thanks to my co-host Ilan. Please send us any feedback you want, whether it’s on the topic, so whether it’s on the podcast itself, and we’ll be more than happy to receive them. Okay, any final words from you guys?
[31:51] Jonny: Cheers.
[31:53] Alex: Cheers.
[31:55] Jonny: You guys stay isolated. And we’ll see you in a couple of weeks.