Functional testing done correctly is very powerful, but it is limited. It verifies that the system is functioning correctly (a very important attribute), but it doesn’t do any verification of usability or appearance.
Enter Web Consistency Testing. The video on the main page gives a very good view into the approach.
Currently, most developer’s develop in their favourite browser until the site looks the expected way. Once that is complete, the site will be checked against another browser and any discrepancies corrected. Continue this process across supported browsers (or wait until someone else notices an issue).
Needless to say this process is very time consuming, prone to errors, and not as thorough as it could be.
Web Consistency Testing is about automating this process. Attempts have been made to do this using image comparison, but depending on site changes (think rotating ads, carousels, browser differences), image comparison can be very fragile. The BBC has written a framework for image comparison that works well for them [elink link="https://github.com/BBC-News/wraith"], but it does require a number of servers, and compares two environments, highlighting differences.
Some problems with this approach include false negatives and difficulty narrowing down the actual discrepancy.
Web Consistency Testing uses the DOM for comparisons. It will parse the DOM, determine locations of items, and use that for comparison. It will also use offsets so that if an item higher in the DOM is out of position, all items below will be offset accordingly so that they are less likely to produce an issue. This really assists in finding an actual discrepancy.
A specific implementation is available at Mogotest. Mogotest leverages a number of open source projects, as well as Cloud servers to evaluate across various browsers, including mobile. It attempts to mimic the developer workflow by using one browser’s rendering as the reference, and then reporting on discrepancies from one run to the next, as well as discrepancies across browsers.
Because it’s leveraging Cloud, it is also able to support many browser/OS combinations, including mobile.
One could duplicate Mogotest’s functionality as it’s primarily built using open source technologies, and readily available Cloud Services.
A very interesting approach to visual verification of a site, and one I will be investigating further.
- Web Consistency Testing: [elink link="http://webconsistencytesting.com/"]
- Mogotest: [elink link="http://mogotest.com/"]