Technological Musings

Thoughts of a technology enthusiast

C9D9 Continuous Delivery panel participation

Agile software development depends on the ability to deploy often, and deploy consistently. How do we end the problem of “it worked on my machine” and ensure fidelity across environments?

On April 21, I enjoyed participating in an online panel on the subject of Consistent Deployments Across Dev, Test and Production, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and Devops.

Continuous Discussions is a community initiative by Electric Cloud, which powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test and deployment processes.

Below are a few soundbites from my contribution in the panel.

How does environment fidelity affect the Continuous Delivery pipeline?

“Without fidelity and control across your environment, I don’t think you can have an effective CD pipeline. The scripts cannot respond to configuration files in random places. If someone can SSH in, it compromises integrity since you do not control what they did. You must have that consistency to automate. You have to have some control over the environment.

“We have to stop treating our servers like puppies. We give them names and we give them all sorts of love. We have to treat them like what they are – they’re just machines. If one of them stops working, you spin up a new one.”

What do you do to ensure fidelity across environments?

“One of the most important things is externalizing your configuration. Even at the basic level, dealing with Java apps and WARs, if your configuration is in your WAR, you have to rebuild that file for every single environment. Then you cannot be sure that your build is repeatable. And if you want to update it with a small configuration change and possibly a server restart, you will have to rebuild all the artifacts. So separating configuration is very important.

“There are many tools for managing configuration: you can go all out and use something like Zookeeper. For small configurations, I have used straight properties files in a shared directory and Spring profiles to manage where it pulls from. For provisioning, I use Chef, Docker, and Puppet. There are many tools to choose from and that’s part of the problem. These tools are always evolving. But they definitely help you move all these things along.

“For keeping the process in line and ensuring you have control over all the servers, destroy the servers periodically. This reminds people that you cannot just go in and make changes, you have to do it the right way or else your changes are not going to be reflected and you will have to do it all over again.

“All of these things feed together. The more you do, the easier your process will be and the more manageable – iteratively, in small steps.”

What if you don’t have fidelity? How do you move from scripting to an end-to-end view of your pipeline and what are the benefits there?

“One of the most important things is getting buy-in from the people you are involved with and ensuring this is something they actually want. Releasing more often can help. One of the primary concepts behind agile is to do it more often. If you do it more often and make it repeatable, it becomes much safer than those big-bang, done-every-six-months releases, which are so dangerous. And then, maybe you’d want to automate the release process.

“As far as moving from scripting to tools – pick a tool that’s going to cover you, start small, pick off some of the easy things, or pick the complicated things because they take a long time and could be turned into a five-minute thing that just gets done. By doing that, people buy into the automation because they see value in it. I think the biggest thing is getting buy-in from the organization, the people, and the customer. Once you have that, it is much easier to justify investing time.

“Try learning tools if you are not familiar with them, or leveraging tools that you already use, and getting people to start using the automation that is put in place, rather than trying to find the quick way to get around it.”

OnePlus phone and Notification issues

I recently acquired the OnePlus phone, and immediately started digging into customizations. One of the features is Quiet Hours.

I didn’t realize it at the time, but Quiet Hours in CyanogenMod 11 u38+ breaks notifications on the OnePlus.

After some Googling, finally found the right post with directions that worked for me (and based on comments, many others). Read the comment by Jake on Oct. 21, 2014 at 8:13 AM.

Here’s a summary:

  • Turn Quiet Hours on
  • Remove the Quiet Hours
  • Restart phone
  • Add Quiet Hours tile back
  • Turn Quiet Hours off using the tile

You may safely remove the Quiet Hours tile after this and until this defect is corrected, don’t turn quiet hours on again.

Floobits: Pair Programming Remotely

Just got the Jetbrains newsletter, and one of the plugins they mentioned is Floobits: Touring Floobits plugin.

Floobits allows for remote sharing of code either through its own built-in editor, or through supported editors. They currently support vim, emacs, Sublime Text and IDEA.

They have a podcast which quickly shows it in action and it looks quite impressive.

If you’re looking to do pair programming with someone, and you’re not in the same place, worth checking out.

Beyond Functional Testing: Web Consistency Testing

Functional testing done correctly is very powerful, but it is limited. It verifies that the system is functioning correctly (a very important attribute), but it doesn’t do any verification of usability or appearance.

Enter Web Consistency Testing. The video on the main page gives a very good view into the approach.

Currently, most developer’s develop in their favourite browser until the site looks the expected way. Once that is complete, the site will be checked against another browser and any discrepancies corrected. Continue this process across supported browsers (or wait until someone else notices an issue).

Needless to say this process is very time consuming, prone to errors, and not as thorough as it could be.

Web Consistency Testing is about automating this process. Attempts have been made to do this using image comparison, but depending on site changes (think rotating ads, carousels, browser differences), image comparison can be very fragile. The BBC has written a framework for image comparison that works well for them, but it does require a number of servers, and compares two environments, highlighting differences.

Some problems with this approach include false negatives and difficulty narrowing down the actual discrepancy.

Web Consistency Testing uses the DOM for comparisons. It will parse the DOM, determine locations of items, and use that for comparison. It will also use offsets so that if an item higher in the DOM is out of position, all items below will be offset accordingly so that they are less likely to produce an issue. This really assists in finding an actual discrepancy.

A specific implementation is available at Mogotest. Mogotest leverages a number of open source projects, as well as Cloud servers to evaluate across various browsers, including mobile. It attempts to mimic the developer workflow by using one browser’s rendering as the reference, and then reporting on discrepancies from one run to the next, as well as discrepancies across browsers.

Because it’s leveraging Cloud, it is also able to support many browser/OS combinations, including mobile.

One could duplicate Mogotest’s functionality as it’s primarily built using open source technologies, and readily available Cloud Services.

A very interesting approach to visual verification of a site, and one I will be investigating further.

Java: Optional.isPresent() is not the best usage of Optional

Using the Optional class from Guava or JDK8?

The TL;DR: If you’re using isPresent(), you’re not taking full advantage of Optional.

Optional allows one to explicitly state that a variable may not have a value rather than relying on comments/code convention and null values.

It provides a number of helpers for constructing Optional items. Optionals are basically a wrapper around an object, and can either be ‘absent’ or ‘present’.

Optional provides an ‘isPresent’ method, but if you’re using it, you aren’t taking full advantage of Optional. Using isPresent isn’t really any different than using {var} != null from a coding perspective.

The Guava Optional provides several methods for retrieving items. If you know it will always have a value (likely due to previous checks), you can use get().

If you’re not sure if it will have a value or not, and want the default returned to be null, use orNull(). There is also an or(defaultValue) that will return defaultValue if it isn’t defined and an or that will return a different Optional value.

The JDK8 version of this is more powerful, but this at least gets one started. Here’s a great post about the JDK8 Optional:

Let's Remember the Real Agile Manifesto

Inflexible Agility. This article is true too often. The focus in agile is not the process, and yet process is talked about more than people, interactions, working software, customer collaboration and responding to change.

Let’s remember those, and not focus on processes. Processes can help, but are secondary.

Groovy for Android Development

Apple recently announced the new programming language for iOS called Swift. One of the languages mentioned by Apple as providing inspiration was Groovy.

Here’s a presentation showing how to use Groovy for Android development. Looks a lot nicer than using Java for it. Take a look and see what you think. I’ll be investigating this for any Android development I have to do.

JUnit: With Java8, do we need Hamcrest anymore?

A very interesting examination of alternative methods for test assertions that leverage lambdas rather than the hamcrest matchers.

Very interesting, and something to keep in mind if you’re fortunate enough to be using Java 8.

Estimation: Story points or story counting?

A very interesting read about estimating and Story Points. Do they truly add value over just counting stories?

In the author’s experience, estimating and Story Points didn’t provide any additional information over counting stories.

There is value in the discussion around stories, and ensuring that stories aren’t too large, but discussing whether it’s a 1, 2, 3 or 5 doesn’t add huge value.

Fallacies that impact our development

An excellent video by Venkat Subramaniam where he discusses how human biases, and the languages we use for development harm us.

Discussions of human nature and conformance, biases and commonly held beliefs that are fallacies.

One example. He asked how many people use Eclipse. He then asked who would pay $500 for Eclipse. Needless to say, most of the participants said no. People pay that much for IntelliJ. Tell you anything about what they think of the IDE? Don’t let free determine your solution, as what else are you potentially giving up?

Another example was effectively ‘no one got fired for buying IBM’. His phrasing was ‘it must be good as it is from a large corporation’. We all have had experiences that disprove that.

All developers are to blame for frameworks getting bloated. A framework exists that solves a problem really well, but hasn’t been updated recently. Instead of still using it, developer’s will seek the item that’s been updated in the last 20 days. So, a product reaches maturity. What’s a company to do to ensure it keeps getting purchased? Enhance it. All we have to do is look at the Office Suites and most other software like that.

Many other examples in here, and as always, some examples of what we put up with coding in Java that we shouldn’t tolerate.

video link: