The Joy of Flex

In a comment, Luke Bayes asked a question about Buzzword’s automated test framework that I thought I should answer a little more prominently.  

Why did we roll our own test framework? Well, one point from Daniel R.’s notes that I’d like to underscore is that we also use FlexUnit extensively to test the Buzzword code that is most amenable to API-style testing. For example, the Document model, which is all about manipulating objects in a tree, is covered pretty well with FlexUnit tests.

But we did build an entirely separate test framework in addition to FlexUnit, and it was a lot of work, so it’s worth explaining more about what we got out of it. More than anything else, what we got out of that work was a test framework that was tightly integrated into Buzzword. This integration allowed us to build features like an interactive test recorder that generates tests as you operate Buzzword, and a step-debugger than lets you step through a recorded test one step at a time to make sure it’s working correctly, while watching the document commands being executed on an open document.

I should add that tight integration between the test framework and the application is not always desirable. In fact, frameworks like FlexUnit are deliberately designed to be more loosely coupled, testing primarily at the API level. This is quite important, but it’s even better to use multiple testing strategies that leverage both tightly coupled and loosely coupled frameworks.

The other question Luke asked was about the pros and cons of relying on ActiveX to support reading and writing local files as part of the test framework, particularly if it prevented running tests against a production environment. This is probably a good time to mention that the primary design goal for our automated test framework was to allow developers to work on a complex shared codebase and verify the correctness of the internal models of the application as we make changes to code. In other words, it was designed to keep us from breaking each others’ code. Because this test framework really only tests the internal models of Buzzword, it is isolated from external environmental factors like OS, browser, or server environment.

One potential disadvantage of this approach is that it means that developers can’t run tests in a Mac development environment. In practice this has not been a major practical issue for us. Most of our developers work under Windows most of the time, and virtually all of our Mac bugs are browser-related. Running these tests isn’t necessary or helpful under the Mac anyway — as with running our tests against a production environment, it wouldn’t provide any new information. 

At this point, you’re probably asking yourself: if this test framework doesn’t test browser-related issues, then how do these guys test against all the different browsers they support.  Good question, and one for another post.