Mean Stack Workflow Automation Yeoman Bower Grunt

width=”100%”—
layout: blogpost
title: Now that’s a MEAN Stack. Workflow Automation – Javascript Style
image: eliran.jpg
author: Eliran Malka
tags:
– DevOps
– Workflow Automation
– Cloud Automation
– UI
– Programming
—

This post was originally posted at Dr. Dobbs.

Labor of love

When we started working on Cloudify’s new version, 3.0 – which is a multi-module project based on Python and YAML, that will have a whole new UI among many other features; as a by-product the dev team was exposed to many new and diverse technologies that required diverse skill sets. Before setting out on this new project, and with agility in mind, we realized that with all this coming-to-life of: more and more new modules in the product, the constant flow of new features and requirements, and the accumulating specs, we’re gonna have to come up with solutions to simplify the project management. We then decided to align our development workflows so that we could better work uniformly across teams and environments. This workflow automation, essentially would enable us to occupy ourselves with the work at hand, rather than the ramp up of getting acquainted with new methodologies each time a new project rolls around, eventually letting us do what we do best: programming.
Since we found ourselves writing new projects for nearly every component in the new product, we were required to cope with various configurations and constellations of projects. Before aligning our workflows, we found ourselves often times spending too much time just becoming familiar with new environments and formations, and with the handover of a project, than the actual tasks at hand. This cost us valuable development time (check back here for a link to an awesome post on other cool hacks for saving developer time), and so all those occupations really needed to be unified in order to help us remain in context and in focus as a team. In this post, I’m going to dive into the process, the workflows we instilled, and the tools we use to get the job done, ultimately to make our dev teams as portable as possible.

Ok, then what’d you do?

To overcome these complexities, we decided to create a unified stack, meaning that every developer on the UI team uses some set of common tools and frameworks (which does not mean we don’t get to learn and implement new and exciting technologies!). For us this basically meant we wanted to use familiar languages, that we know best (yes, we love JavaScript), while automating away all the nitty gritty involved with bootstrapping projects, boilerplate code etc.  So, ultimately, all those usually tedious jobs become effortless, and we form intuitive habits in our work process. Habits are important, and breaking habits interferes with the flow of thought.

Cloudify 3.1 – workflow automation for cloud apps made easy. Give it a whirl..  Go

We unified the stack by choosing the most popular technologies that are tried and well tested, but most importantly, are also active, as active projects mean stable code. For the purpose of aligning our project structures and automatically generating code we chose to work with Yeoman. For building and packaging we decided to go with Grunt.  Bower and NPM then provide ready-made components for the frontend and backend, respectively. As for the actual development of the product, we use AngularJS, NodeJS and Express almost exclusively, although some N-Tier applications (like the re-implemented Cloudify Player) are even going further, utilizing a full MEAN stack. Since, the motivation is to use familiar technology, and because MongoDB accepts JSON, the use of MongoDB allows us to go JavaScript all the way (habits are important, remember?).

Your workflow, how goes it?

Well, having chosen this stack, the first thing a developer would do to create a new project, is go to the command line and type yo angular:app. This tells Yeoman to auto-generate the full project structure for an AngularJS client, including in-place configuration files for build and package management. We do the same for other layers, and we rely on out-of-the-box code generators to eliminate friction when updating these generation tools. With this common boilerplate, we can know that all of our projects will look the same, therefore, on top of providing us the benefit of familiarity with JavaScript-based technologies, there is also the added benefit of familiarity with the project structure.  This is very important for context and a quick ramp up. Once you get used to it, it becomes very natural to work with.
After you’ve built your project (or cloned it), you then need to do two things; fetch external dependencies and assemble it all together. To handle dependencies in our environment, we use the NodeJS Package Manager to include third-party libraries for the backend, and Bower that manages client-side components. As Bower, NPM and Grunt already have their own configuration files included in each project, this means that after we’ve decided on our components, and defined them in these configuration files, we then simply call an NPM install (which, surprisingly, involves running npm install in the command line), a Bower install (guess how?), and then call a Grunt build (well, you know the drill) and POOF, all of our components are included and injected into the project automagically (yeah, I hate that word, too). This lets us immediately start working.

Ready made components, you say?

Yes. We are developers. We hate to reinvent the wheel. So we use what other developers, like us, have written in the past to solve issues just like ours. For instance, if a developer needs some sort of a slider component to be used in a web page, she can just look up the bower repository with bower search, find an appropriate slider, let’s say angular-slider, and then all she’d have to do is bower install angular-slider – and it’s integrated into the project.
To be honest, some of the time, those developers that write components are ourselves. We have our very own Kitchen Sink of reusable components we built for generic use, that we can use in any project in the exact same manner – they’re all published into Bower!

Neat. How do you run it?

Glad you asked. Well, while developing in the client, we use preview servers – in our case provided by Grunt’s LiveReload and Watch plugins, so we really don’t have to refresh the browser page at any point – which is annoying for developers worldwide.
Once Grunt’s development server is up and running, it will watch for any change in the project files and reload that specific resource in the web-app using websockets. This allows us to work continuously, Grunt rewires the app into the browser as we go along, and no more clicking the refresh button. Ever. It’s all very neat.

Ok, what about integration?

Once we finished developing a certain feature, the next thing to do is to commit it into the source control, but before the check-in, it needs to be validated and tested. Again we use Grunt, that provides some helpers out of the box. To verify our code validates (JavaScript is an interpreter language, so it does not compile) we go to the command line, type grunt jshint, and it runs validation on the code. To run tests, just grunt test. Well, actually we only need to run a single command, which is simply grunt, that does it all for us. Once it passes locally, you’re ready to commit to Github, and have the CI tools do their job. Needless to say the CI tools will also be using Grunt to create the distribution (dist is the Grunt task responsible for that).
We write tests with Jasmine, which is a behavior-driven development framework for testing JavaScript code, and the nice thing is that an entire test suite scaffold is written for us the moment we add some auto-generated code. Yeoman does that for us, and all we have to do is fill in the test-methods bodies.
There’s a lot to be said about continuous integration, that I won’t dive into here (which you can find more on in another excellent post), but I’ll just share that we use TravisCI and QuickBuild to manage our tailored/nightly builds.

Nice. Is that it?

Nope. There are also profiles! A profile means coloring an environment with certain configuration flavor, for a specific need. When developing, our environment has two flavors of configuration, one for production, and the other for development – which can also be used for testing. This separation provides us with convenient mocking abilities when testing or developing (e.g. why use a full-blown backend server when all you need is a fixed response? Mock it up with Jasmine!) and also protects us from incorporating any sensitive data into the production code and possibly have it checked-in into the repository.  These include passwords, secret keys, API keys and so forth. Those all reside in the development profile, while the production profile will be built with the real credentials. Again, once developers form a habit of working with profiles, it becomes second nature, and they don’t even think about doing it. Just agree on a private place for your personal configuration files, switch the dev profile on, and roll with it.

Post scriptum

When we, as developers, started using unified workflows and tools, we became highly portable were able to hop from one team to another easily, and work seamlessly across diverse projects, always keeping it fresh and interesting. From developing a UI component, via working on a feature at the backend, well into writing automated tasks and so forth. This means we can work intuitively and concentrate on the important things, increasing productivity and speeding up delivery of projects exponentially. And there was much rejoicing.
At the end of the day, aligning workflows means less friction. Less friction means less worries. Less worries means more coding, and more coding means happy developers. Well at least that’s the case with me. I love coding.