Agile software development depends on the ability to deploy often, and deploy consistently. How do we end the problem of “it worked on my machine” and ensure fidelity across environments?
On April 21, I enjoyed participating in an online panel on the subject of Consistent Deployments Across Dev, Test and Production, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and Devops.
Continuous Discussions is a community initiative by Electric Cloud, which powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test and deployment processes.
Below are a few soundbites from my contribution in the panel.
How does environment fidelity affect the Continuous Delivery pipeline?
“Without fidelity and control across your environment, I don’t think you can have an effective CD pipeline. The scripts cannot respond to configuration files in random places. If someone can SSH in, it compromises integrity since you do not control what they did. You must have that consistency to automate. You have to have some control over the environment.
“We have to stop treating our servers like puppies. We give them names and we give them all sorts of love. We have to treat them like what they are – they’re just machines. If one of them stops working, you spin up a new one.”
What do you do to ensure fidelity across environments?
“One of the most important things is externalizing your configuration. Even at the basic level, dealing with Java apps and WARs, if your configuration is in your WAR, you have to rebuild that file for every single environment. Then you cannot be sure that your build is repeatable. And if you want to update it with a small configuration change and possibly a server restart, you will have to rebuild all the artifacts. So separating configuration is very important.
“There are many tools for managing configuration: you can go all out and use something like Zookeeper. For small configurations, I have used straight properties files in a shared directory and Spring profiles to manage where it pulls from. For provisioning, I use Chef, Docker, and Puppet. There are many tools to choose from and that’s part of the problem. These tools are always evolving. But they definitely help you move all these things along.
“For keeping the process in line and ensuring you have control over all the servers, destroy the servers periodically. This reminds people that you cannot just go in and make changes, you have to do it the right way or else your changes are not going to be reflected and you will have to do it all over again.
“All of these things feed together. The more you do, the easier your process will be and the more manageable – iteratively, in small steps.”
What if you don’t have fidelity? How do you move from scripting to an end-to-end view of your pipeline and what are the benefits there?
“One of the most important things is getting buy-in from the people you are involved with and ensuring this is something they actually want. Releasing more often can help. One of the primary concepts behind agile is to do it more often. If you do it more often and make it repeatable, it becomes much safer than those big-bang, done-every-six-months releases, which are so dangerous. And then, maybe you’d want to automate the release process.
“As far as moving from scripting to tools – pick a tool that’s going to cover you, start small, pick off some of the easy things, or pick the complicated things because they take a long time and could be turned into a five-minute thing that just gets done. By doing that, people buy into the automation because they see value in it. I think the biggest thing is getting buy-in from the organization, the people, and the customer. Once you have that, it is much easier to justify investing time.
“Try learning tools if you are not familiar with them, or leveraging tools that you already use, and getting people to start using the automation that is put in place, rather than trying to find the quick way to get around it.”