Improving Your Process: Automated Regression Testing with Fake.app

Testing is a really important part of writing software. Recently, I’ve been increasingly exposed to regression testing, something I haven’t really needed to do much lately. At the surface, regression testing is a technique for testing software aimed at bringing to light any new errors (or regressions) in a piece of code after changes have been made. Regression testing is an important part of subsequent version releases as it ensures nothing that was working before will be broken in the new version.

Something I plan on focusing on to a great extent in 2012 is upping the overall quality of software I’m writing, a big chunk of that will be finalizing a testing plan for Iron to Iron projects. Regression testing will be a really big part of that, and I’d like to share my game plan.

A recent Attachments Pro release included a regression that I should have caught. The plugin has expanded to be big enough where I unknowingly got lazy with testing prior to what I thought was a quick bug fix release. Having a regression slip through the cracks was enough for me to sit down and truly evaluate how I could prevent it from happening again, so I took a look at what was available.

Regression testing is often a very manual process. You follow a regression document that outlines the steps you should take to use a piece of software (e.g. “click this, do that, enter this text, click submit”). With each step you should evaluate if the software is behaving properly, and note any bugs you find. As the software becomes more advanced, as does the regression document, resulting in a very tedious process. We as humans will quickly tire of regression testing, often resulting in cut corners and overlooked bugs, as was my case recently with Attachments Pro.

Unit testing, by contrast, is a very automated way of testing software. You write software to test your software and ensure expected results are obtained. I wanted to establish a regression testing method for Attachments Pro that more closely represented unit testing, not only to make my life easier, but to ensure no corners were cut.

I first tried to estimate how long it would take me to build a test suite and what that would involve. It only took me a few minutes to realize that it was a project unto itself and not something I’d be able to pull off any time soon, especially with a list of things I’d like to take care of with Attachments Pro.

Enter Fake.app

Much to my surprise, I had forgotten that what I was looking to do had already been done! Todd Ditchendorf (creator of Fluid.app and a ton of other awesome Web tools) solved this problem for us by combining the browser and Automater.app. For those that don’t use os x or aren’t familiar, Automater.app pretty much does what’s on the label; automate things. It’s an extremely elaborate application meant to alleviate the tediousness of various desktop operations you may want to run. Fake brings that luxury to a browser.

Todd approached this project just right in that the issue is far from simple, but he kept the complexity to a level where exposing it is a product of your own explanation.

The beauty lies within actually creating workflows and watching them run. Workflows can be as simple, complex, or recursive as you want and to put it simply my mind was blown the first time I built an applicable test for Attachments Pro.

I’m in the process of building and refining my test suite for Attachments Pro, and I’m truly looking forward to fully integrating Fake.app into my deployment process. I can’t help but to gush over the way Todd has built Fake.app; he’s literally thought of everything and implemented it in such a way that building a workflow is actually fun. The tools pull data when they can, making option-choosing that much easier, and the entire interaction process when working with Fake is just a pleasure to use. It stays out of your way and lets you get the job done.

One of my favorite parts of Fake is the way you’re able to pick and choose elements to work with. For instance, much of the work I’ll be doing with Fake is form submitting. I was very curious how Fake would allow me to work with something like Attachments Pro since so many form elements are generated dynamically. Todd solved that by allowing every possible way of selecting elements to work with. For example, by using the CSS Selector option, you’re able to use pseudo-selectors to grab the last-of-type which allows me to work with the most recently created set of form elements. Voilà!

Those touches are what really sold me on Fake.app for automated regression testing, I can’t wait to finalize my test suite and directly integrate the workflow into my deployment process. Not only will it take less time and avoid human mistakes, it’ll help me build more solid software from here on out.

Windows Alternatives

It’s always difficult to write something based on a product limited to a single platform, especially when it comes to os x and windows, but at the same time there are platform products that can do wonders for a workflow and are worth singling out.

Luckily in this case there are options beyond os x, albeit not for Fake exactly. If you’re interested, have a look at Selenium, which touts itself as “… a suite of tools to automate web browsers across many platforms”. I had a quick look at Selenium some time ago and was in part confused as to how the whole system worked, let alone what pieces I needed to download, configure, and get working. That said, I haven’t explored Selenium to any extent, but would be interested in hearing from someone who has.

Feature Requests for Fake

Now that I’ve had the luxury or working with Fake and beginning my Attachments Pro regression test suite, there are a couple of things I’d love to put on a wish list. I recognize that regression is likely too niche and far beyond Todd’s original plan for the application, but with the addition of a few things, Fake would be that much better at automating a test suite.

Fake has variable definitions, but I’d love for that to be a bit more integrated throughout. For instance, I’d love to set a variable to either a static (i.e. defined when writing the workflow) or dynamic (i.e. pulled from either a JavaScript or field value on page during ‘runtime’) and them have a reference to that variable with Fake’s already present conditionals. Adding that would ad the ability to check any number of conditions at any time throughout the workflow and give even more opportunity to uncover a regression. As it stands, Fake offers a ton when it comes to options for conditionals, but a reference to a created variable hasn’t been implemented.

The only other feature request I can come up with that would help a lot with regression testing specifically would be the generation of random data (e.g. for form field population). It would be completely stellar if you could generate random data based on a regex pattern somehow. For example, Attachments Pro requires the name of each instance to be unique. I would love to have a workflow that could generate any number of instances, each with a unique name that could be generated on the fly. This feature request is one of those complete edge cases, but it’s one of the two I could come up with for Fake.

All in all, I’d love to encourage you to check out Fake.app (or Selenium) and determine if/how you can incorporate it into your workflow. Unit testing is awesome, but I truly believe solutions like Fake have their place in regression testing.