Test Tools

(There has been news coverage recently on the number of blogs started and then abandoned - something like two million, if memory serves. Tied to the coverage, a new term was coined to describe people who start blogs and abandon them - cloggers. So, brought to you courtesy of a vague sense of guilt, a new entry.)

Very few of the tools I use for testing are specifically designed for testing. The honourable exceptions are defect reporting tools and test management software, but in both of those cases I would be (and have been) happy using a spreadsheet or some other general tool in certain circumstances.

The reason I use so few specialised testing tools is probably because most of my background has been on UNIX systems, where there are many powerful general tools that can be easily manipulated and strung together to produce tools for specific tasks. I also have a suspicion that specialised testing tools are hugely over-priced, and either solve problems that I don't need to solve or are so generalised that the effort required to configure them to do what I need to do is larger than that needed for me to write a tool myself in Perl. *

For the first time ever in my career, I am now working on a product that is installed onto a normal user PC running Windows. And I have found a wonderful, non-testing-specific tool that has found several bugs that I probably wouldn't have found without it, and helped diagnose countless others. Process Explorer, distributed free by what used to be sysinternals.com, is basically a souped up version of Task Manager, or a Windows GUI version of the top command. **

I can see what processes are running, when they started, and what started them. I can see CPU usage, RAM consumption, Virtual memory usage, and IO history for the whole system, or for each individual process. I can see what threads each process is running, and the stack trace on each of those threads.

I keep Process Explorer running on a second monitor, where I can see it out of the corner of my eye. New processes starting are coloured green, and processes terminating are coloured red (they can be changed, but I like the defaults), and this will catch my attention. I can see at a glance whether the amount of RAM has risen or fallen significantly, whether any particular process is using the CPU when it shouldn't be working, and basically where the activity is coming from.

But where it has proven most valuable is in noticing when processes haven't terminated when they should. Instead of finding these processes during a rare glance at Task Manager, when I have no idea when they started and what I was doing at the time, I will see it as it happens, and be able to investigate the bug while my previous actions are fresh in my mind.

And this feeds into something I believe about test case design and its value. I could have defined one test case (or several, or an infinity of test cases) that checked whether all processes terminated when they should have after executing various user scenarios. And I would have spent a few days working through those scenarios, driving myself half crazy with the tedium, and perhaps halfway through I would have noticed that I was running short on RAM, and I would start again to see which scenario was to blame, and a whole heap of time would have been taken up executing a test case that was better executed without being defined, while executing other test cases.

It isn't just with these kinds of tests, monitoring system behaviour, that I think leaving test cases initially undefined is a good idea, but that's a huge topic for another day.

* In fact, this is blatantly untrue. Open Source Testing has a lot of free resources listed, and although many of them don't solve my problems, some of them have proven useful and reduced my workload. If I can't overcome my prejudices, at least I can be aware of what they are.

** As implied, I have very little experience on Windows systems, and can just about navigate the file system in DOS. Although I tell this to anybody who listens, they seldom believe me. So while there may be DOS alternatives to do what I use Process Explorer to do, I didn't find information on them.