Testing is a really important part of writing software. Recently, I’ve been increasingly exposed to regression testing, something I haven’t really needed to do much lately. At the surface, regression testing is a technique for testing software aimed at bringing to light any new errors (or regressions) in a piece of code after changes have been made. Regression testing is an important part of subsequent version releases as it ensures nothing that was working before will be broken in the new version.
Something I plan on focusing on to a great extent in 2012 is upping the overall quality of software I’m writing, a big chunk of that will be finalizing a testing plan for Iron to Iron projects. Regression testing will be a really big part of that, and I’d like to share my game plan.
A recent Attachments Pro release included a regression that I should have caught. The plugin has expanded to be big enough where I unknowingly got lazy with testing prior to what I thought was a quick bug fix release. Having a regression slip through the cracks was enough for me to sit down and truly evaluate how I could prevent it from happening again, so I took a look at what was available.
Regression testing is often a very manual process. You follow a regression document that outlines the steps you should take to use a piece of software (e.g. “click this, do that, enter this text, click submit”). With each step you should evaluate if the software is behaving properly, and note any bugs you find. As the software becomes more advanced, as does the regression document, resulting in a very tedious process. We as humans will quickly tire of regression testing, often resulting in cut corners and overlooked bugs, as was my case recently with Attachments Pro.
Unit testing, by contrast, is a very automated way of testing software. You write software to test your software and ensure expected results are obtained. I wanted to establish a regression testing method for Attachments Pro that more closely represented unit testing, not only to make my life easier, but to ensure no corners were cut.
I first tried to estimate how long it would take me to build a test suite and what that would involve. It only took me a few minutes to realize that it was a project unto itself and not something I’d be able to pull off any time soon, especially with a list of things I’d like to take care of with Attachments Pro.
Much to my surprise, I had forgotten that what I was looking to do had already been done! Todd Ditchendorf (creator of Fluid.app and a ton of other awesome Web tools) solved this problem for us by combining the browser and Automater.app. For those that don’t use os x or aren’t familiar, Automater.app pretty much does what’s on the label; automate things. It’s an extremely elaborate application meant to alleviate the tediousness of various desktop operations you may want to run. Fake brings that luxury to a browser.
Todd approached this project just right in that the issue is far from simple, but he kept the complexity to a level where exposing it is a product of your own explanation.
The beauty lies within actually creating workflows and watching them run. Workflows can be as simple, complex, or recursive as you want and to put it simply my mind was blown the first time I built an applicable test for Attachments Pro.
I’m in the process of building and refining my test suite for Attachments Pro, and I’m truly looking forward to fully integrating Fake.app into my deployment process. I can’t help but to gush over the way Todd has built Fake.app; he’s literally thought of everything and implemented it in such a way that building a workflow is actually fun. The tools pull data when they can, making option-choosing that much easier, and the entire interaction process when working with Fake is just a pleasure to use. It stays out of your way and lets you get the job done.
One of my favorite parts of Fake is the way you’re able to pick and choose elements to work with. For instance, much of the work I’ll be doing with Fake is form submitting. I was very curious how Fake would allow me to work with something like Attachments Pro since so many form elements are generated dynamically. Todd solved that by allowing every possible way of selecting elements to work with. For example, by using the CSS Selector option, you’re able to use pseudo-selectors to grab the
last-of-type which allows me to work with the most recently created set of form elements. Voilà!
Those touches are what really sold me on Fake.app for automated regression testing, I can’t wait to finalize my test suite and directly integrate the workflow into my deployment process. Not only will it take less time and avoid human mistakes, it’ll help me build more solid software from here on out.
It’s always difficult to write something based on a product limited to a single platform, especially when it comes to os x and windows, but at the same time there are platform products that can do wonders for a workflow and are worth singling out.
Luckily in this case there are options beyond os x, albeit not for Fake exactly. If you’re interested, have a look at Selenium, which touts itself as “… a suite of tools to automate web browsers across many platforms”. I had a quick look at Selenium some time ago and was in part confused as to how the whole system worked, let alone what pieces I needed to download, configure, and get working. That said, I haven’t explored Selenium to any extent, but would be interested in hearing from someone who has.
Feature Requests for Fake
Now that I’ve had the luxury or working with Fake and beginning my Attachments Pro regression test suite, there are a couple of things I’d love to put on a wish list. I recognize that regression is likely too niche and far beyond Todd’s original plan for the application, but with the addition of a few things, Fake would be that much better at automating a test suite.
The only other feature request I can come up with that would help a lot with regression testing specifically would be the generation of random data (e.g. for form field population). It would be completely stellar if you could generate random data based on a regex pattern somehow. For example, Attachments Pro requires the name of each instance to be unique. I would love to have a workflow that could generate any number of instances, each with a unique name that could be generated on the fly. This feature request is one of those complete edge cases, but it’s one of the two I could come up with for Fake.
All in all, I’d love to encourage you to check out Fake.app (or Selenium) and determine if/how you can incorporate it into your workflow. Unit testing is awesome, but I truly believe solutions like Fake have their place in regression testing.
I’m curious, what steps do you take now in terms of unit testing and integration testing? How does this work in your ecosystem (PHP/Wordpress)?
Regression testing is good, but I am wondering what steps you take earlier – or what practices you do earlier in the TDD process.
Testing your UI is important, but I think this decoupling into an app is scary for a few reasons.
1. As your code evolves, you have to ensure the external tests are always up to date with the current codebase.
2. Distribution. Could another developer get your code from github and run your test suite, modify the code with confidence, and push back to a repository? Do you keep this test/execution with the codebase itself?
Any discussion about testing is a good one 🙂 I actually like what I see here, just curious about the impacts on an application ecosystem.
Great questions, Nate! As it stands, I honestly don’t have a formal TDD process for WordPress itself. When it comes to WordPress core, there are docs for formal testing available, so I don’t repeat those tests when I’m simply building a custom theme. If I do build a significant addition to a site for a client though, I’ll need to write my own tests for that bit.
Decoupling is a major concern, and I’m glad you’ve raised it — I’ll show my true colors here in explaining that testing is not something I’ve been widely exposed to, so this is one of those cases where I’m not sure how to avoid the need for tests needing to be kept up to date with the codebase, especially when it comes to testing the UI side of things. Any insight on alternative methods here would be stellar!
When it comes to Attachments Pro, I wouldn’t be distributing the tests or the codebase for that matter. That aside, I think we can keep your original purpose in mind: how would this work for, say, Attachments? I think it goes without saying that locking the tests into something like Fake could be considered a drawback. Not only is it an external app, but it’s a commercial piece of software limited to a particular environment. It might not be the best solution for an open source project, but if you’re developing software in house I think considering it as part of your testing is worthwhile.
I’d love to hear more feedback in particular from your side of things though, Nate, we’ve definitely had our back-and-forths about WordPress (and PHP in general), but I’d love to hear more about the details behind the software you’re building and how you test it!