Agile software development depends on the ability to deploy often, and deploy consistently. How do we end the problem of “it worked on my machine” and ensure fidelity across environments?
On April 21, I enjoyed participating in an online panel on the subject of Consistent Deployments Across Dev, Test and Production, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and Devops.
Continuous Discussions is a community initiative by Electric Cloud, which powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test and deployment processes.
Below are a few soundbites from my contribution in the panel.
How does environment fidelity affect the Continuous Delivery pipeline?
“Without fidelity and control across your environment, I don’t think you can have an effective CD pipeline. The scripts cannot respond to configuration files in random places. If someone can SSH in, it compromises integrity since you do not control what they did. You must have that consistency to automate. You have to have some control over the environment.
“We have to stop treating our servers like puppies. We give them names and we give them all sorts of love. We have to treat them like what they are – they’re just machines. If one of them stops working, you spin up a new one.”
What do you do to ensure fidelity across environments?
“One of the most important things is externalizing your configuration. Even at the basic level, dealing with Java apps and WARs, if your configuration is in your WAR, you have to rebuild that file for every single environment. Then you cannot be sure that your build is repeatable. And if you want to update it with a small configuration change and possibly a server restart, you will have to rebuild all the artifacts. So separating configuration is very important.
“There are many tools for managing configuration: you can go all out and use something like Zookeeper. For small configurations, I have used straight properties files in a shared directory and Spring profiles to manage where it pulls from. For provisioning, I use Chef, Docker, and Puppet. There are many tools to choose from and that’s part of the problem. These tools are always evolving. But they definitely help you move all these things along.
“For keeping the process in line and ensuring you have control over all the servers, destroy the servers periodically. This reminds people that you cannot just go in and make changes, you have to do it the right way or else your changes are not going to be reflected and you will have to do it all over again.
“All of these things feed together. The more you do, the easier your process will be and the more manageable – iteratively, in small steps.”
What if you don’t have fidelity? How do you move from scripting to an end-to-end view of your pipeline and what are the benefits there?
“One of the most important things is getting buy-in from the people you are involved with and ensuring this is something they actually want. Releasing more often can help. One of the primary concepts behind agile is to do it more often. If you do it more often and make it repeatable, it becomes much safer than those big-bang, done-every-six-months releases, which are so dangerous. And then, maybe you’d want to automate the release process.
“As far as moving from scripting to tools – pick a tool that’s going to cover you, start small, pick off some of the easy things, or pick the complicated things because they take a long time and could be turned into a five-minute thing that just gets done. By doing that, people buy into the automation because they see value in it. I think the biggest thing is getting buy-in from the organization, the people, and the customer. Once you have that, it is much easier to justify investing time.
“Try learning tools if you are not familiar with them, or leveraging tools that you already use, and getting people to start using the automation that is put in place, rather than trying to find the quick way to get around it.”
Functional testing done correctly is very powerful, but it is limited. It verifies that the system is functioning correctly (a very important attribute), but it doesn’t do any verification of usability or appearance.
Currently, most developer’s develop in their favourite browser until the site looks the expected way. Once that is complete, the site will be checked against another browser and any discrepancies corrected. Continue this process across supported browsers (or wait until someone else notices an issue).
Needless to say this process is very time consuming, prone to errors, and not as thorough as it could be.
Web Consistency Testing is about automating this process. Attempts have been made to do this using image comparison, but depending on site changes (think rotating ads, carousels, browser differences), image comparison can be very fragile. The BBC has written a framework for image comparison that works well for them https://github.com/BBC-News/wraith, but it does require a number of servers, and compares two environments, highlighting differences.
Some problems with this approach include false negatives and difficulty narrowing down the actual discrepancy.
Web Consistency Testing uses the DOM for comparisons. It will parse the DOM, determine locations of items, and use that for comparison. It will also use offsets so that if an item higher in the DOM is out of position, all items below will be offset accordingly so that they are less likely to produce an issue. This really assists in finding an actual discrepancy.
A specific implementation is available at Mogotest. Mogotest leverages a number of open source projects, as well as Cloud servers to evaluate across various browsers, including mobile. It attempts to mimic the developer workflow by using one browser’s rendering as the reference, and then reporting on discrepancies from one run to the next, as well as discrepancies across browsers.
Because it’s leveraging Cloud, it is also able to support many browser/OS combinations, including mobile.
One could duplicate Mogotest’s functionality as it’s primarily built using open source technologies, and readily available Cloud Services.
A very interesting approach to visual verification of a site, and one I will be investigating further.