In my first post, I mentioned that I was far too busy outside of work - one of the things that I do outside of work is amateur dramatics, and last weekend I was stage managing a Shakespearean production in the old opera house in Helsinki.
I enjoy acting, and always have done, but recently I've come to appreciate working backstage instead. In part, it's because I like being in control, but also it is a completely different way of using the same skills I use in the office.
People look at me as if I'm insane when I say that, but it's true.
Observation: during rehearsals I am watching everything that goes on, from which of the actors chatter all the time, to who isn't ready for their entrance, to the magically vanishing prop (the actor is miming holding a cup right up until the moment they want to use both hands, and hey presto the imaginary cup is whisked out of existence). I watch for traffic jams, when too many people leave the stage at the same time, and run into people coming on stage.
During performances, I watch even closer. The final performance of our Shakespearean play, a piece of something floated down onto the stage from the flies. I needed to find a way to get it off the stage as soon as possible, but I also needed to find out what it was - we had an enormous contraption suspended above the stage, and if it was falling to pieces we would be in big trouble.
There was an easy solution to the first part, we had a scene change coming up and I could pick up the something en route, which turned out to be a non-essential bit of duct tape (which, incidentally, I should never have left behind in the first place - a mistake I won't be making again), but if I hadn't been looking that might have stayed on the stage for the rest of the evening. I could also reassure the few actors who noticed it that they weren't about to get any heavy surprises falling on their heads.
So, as suggested above, all my observation is for the purpose of anticipating problems before they occur. Loose edges of mats can be tripped over in the dark, loose cables can be dragged across the floor, unshielded lamps in the wings can flood onto the stage during blackouts, dropped props have to be taken off stage before the next scene begins, scenery needs to be put in the same place every time or the lighting will be wrong, and so on. The number of things that can go wrong backstage is huge, and I need to identify as many as possible. I also need to find solutions to the problems, and then implement the solutions, which is where the analogy to testing falls down (somewhat, if you think of the problems that I anticipate as bugs).
In a large production like our Shakespeare, I am not alone but leading a team who also look for problems and solve them. My Assistant Stage Manager came up with the idea for suspending our contraption above the stage after anticipating problems moving it on and off stage in a blackout, making no noise. One of my stage hands took on the problem of two slightly unruly actors who had to help in a scene change, and sorted them out so that they wouldn't collide with anybody or get in the way. An experienced actor impressed upon a twelve-year-old member of the cast the importance of not standing in front of the lighting rack in the wings, casting her shadow across the entire stage.
I don't want to push the analogy too far. In the office, somebody moving my tools from where I left them is irritating - backstage, my anger was incandescent when somebody (and nobody will own up to it) moved my roll of duct tape during performance. If I hadn't checked for it before I needed it, we would have had quite a problem.
One thing that I bring from stage management back to testing is that many problems can be solved with very low-tech methods. Sure, suspending a contraption over the stage to be lowered into place is pretty high-tech, but another tricky problem was solved with three bottles of water, some crumpled up newspaper and a napkin. So in testing, while to execute one test I might use packet sniffing software, a website mirroring tool and a testing interface to the test item, to execute another I use maybe a vi macro, or a three-line Perl script. The thing that matters is how well the solution solves the problem, not how technically advanced the solution is.
12 November 2006
30 October 2006
My first bug
I found my first bug on my first day at the Giant Fruit Company. It was a crash bug.
I mention this in interviews a lot, as I think it sounds cool. At the time, it made me think that I could get into this "testing thing", but there is a better lesson to be learned from how and why I found it.
As I mentioned, I started working at the Giant Fruit Company testing the porting of existing software onto their range of laptops. My first day, after introductions, I was given the task of testing Microsoft Word on the new laptop. For this kind of system, we had two guidelines. One was for testing the things that all applications were supposed to do on the Giant Fruit Company's operating system, in other words the GUI consistency. The second was for testing the behaviour of the particular application itself. In current jargon, they were test oracles. Altogether we spent ten hours on each application, five or six hours following the guidelines, the rest of the time on exploratory testing, where we just used the application to complete a task it would normally be used for.
My mentor left for a meeting, leaving me armed with the guidelines, a prototype laptop with correct installation, and some excitement at My Very First Testing Attempt.
Twenty minutes later I had crashed the laptop. I had a moment of panic, before remembering that in my new career this was actually a Good Thing. I carefully wrote down what I had done, then rebooted and checked whether I could do it again. I could! But what if I was doing it wrong?
My mentor returned from his meeting, and I started into my preamble. I had got to this point in the first guideline, and double-clicked on the menu bar, and the result wasn't what it was supposed to be.
"Oh yeah," he started. "That test is out of date, they got rid of that functionality a couple of releases back. Just skip it."
"But," I told him, anxiously. And showed him I crashed the laptop by carrying on double-clicking on the menu bar.
For each bug, we had a set regression path to follow, up through previous laptop incarnations until we found where the bug was introduced. This one had been with range since its very start. It remained unknown until I, as a complete newbie, stumbled on it.
From this, I got two lessons:
One: fresh eyes and ignorance are good things
Two: when bored, break things
Okay, I'm cheating a bit. I broke it because I was confused, there was a bug in my test oracle, and repeating the test until it worked was the only thing I could think of doing. But the idea of carrying on doing unexpected things was valuable, and I've found a lot of bugs on that principle alone. It's something I do when I feel bored, when dutifully verifying that x variation of test dimension y becomes too dull. There are always bugs like that to find.
Sadly, a couple of months later I learned a third lesson from that bug:
Three: large companies aren't necessarily structured so that bugs get fixed
When I left the Giant Fruit Company, that bug was still unfixed. The test team had one point of contact outside the test organisation, somebody called the Quality Lead. It became obvious that the Quality Lead was being evaluated on the number of bugs open on his project domain in the database. By labeling the bug as "Third party problem", it moved out of his domain and became somebody else's problem. And since it was a problem found with third party software, his rule of thumb was that it was a third party problem - even though outside this range of laptops it didn't occur.
Can you see where this is leading? We spent a large part of our time testing how third party applications behaved on our new laptop. Any bugs we found were deemed to be third party problems and never fixed, even when they clearly weren't. Morale was pretty low. Bug reports started getting abusive, people were fired for abusive bug reports, change was promised but never quite happened.
Most of the problem? We never even met the Quality Lead, never mind the people actually responsible for developing the new laptop. They were based on the West Coast US, we were in Ireland, we had phone conferences with other testers but that was about it. This did actually begin to change before I left, but too little too late.
I mention this in interviews a lot, as I think it sounds cool. At the time, it made me think that I could get into this "testing thing", but there is a better lesson to be learned from how and why I found it.
As I mentioned, I started working at the Giant Fruit Company testing the porting of existing software onto their range of laptops. My first day, after introductions, I was given the task of testing Microsoft Word on the new laptop. For this kind of system, we had two guidelines. One was for testing the things that all applications were supposed to do on the Giant Fruit Company's operating system, in other words the GUI consistency. The second was for testing the behaviour of the particular application itself. In current jargon, they were test oracles. Altogether we spent ten hours on each application, five or six hours following the guidelines, the rest of the time on exploratory testing, where we just used the application to complete a task it would normally be used for.
My mentor left for a meeting, leaving me armed with the guidelines, a prototype laptop with correct installation, and some excitement at My Very First Testing Attempt.
Twenty minutes later I had crashed the laptop. I had a moment of panic, before remembering that in my new career this was actually a Good Thing. I carefully wrote down what I had done, then rebooted and checked whether I could do it again. I could! But what if I was doing it wrong?
My mentor returned from his meeting, and I started into my preamble. I had got to this point in the first guideline, and double-clicked on the menu bar, and the result wasn't what it was supposed to be.
"Oh yeah," he started. "That test is out of date, they got rid of that functionality a couple of releases back. Just skip it."
"But," I told him, anxiously. And showed him I crashed the laptop by carrying on double-clicking on the menu bar.
For each bug, we had a set regression path to follow, up through previous laptop incarnations until we found where the bug was introduced. This one had been with range since its very start. It remained unknown until I, as a complete newbie, stumbled on it.
From this, I got two lessons:
One: fresh eyes and ignorance are good things
Two: when bored, break things
Okay, I'm cheating a bit. I broke it because I was confused, there was a bug in my test oracle, and repeating the test until it worked was the only thing I could think of doing. But the idea of carrying on doing unexpected things was valuable, and I've found a lot of bugs on that principle alone. It's something I do when I feel bored, when dutifully verifying that x variation of test dimension y becomes too dull. There are always bugs like that to find.
Sadly, a couple of months later I learned a third lesson from that bug:
Three: large companies aren't necessarily structured so that bugs get fixed
When I left the Giant Fruit Company, that bug was still unfixed. The test team had one point of contact outside the test organisation, somebody called the Quality Lead. It became obvious that the Quality Lead was being evaluated on the number of bugs open on his project domain in the database. By labeling the bug as "Third party problem", it moved out of his domain and became somebody else's problem. And since it was a problem found with third party software, his rule of thumb was that it was a third party problem - even though outside this range of laptops it didn't occur.
Can you see where this is leading? We spent a large part of our time testing how third party applications behaved on our new laptop. Any bugs we found were deemed to be third party problems and never fixed, even when they clearly weren't. Morale was pretty low. Bug reports started getting abusive, people were fired for abusive bug reports, change was promised but never quite happened.
Most of the problem? We never even met the Quality Lead, never mind the people actually responsible for developing the new laptop. They were based on the West Coast US, we were in Ireland, we had phone conferences with other testers but that was about it. This did actually begin to change before I left, but too little too late.
Welcome - why are we here at all?
Welcome to my new blog, where the main focus will be on my professional life as a software tester.
I decided to start blogging about testing for a host of reasons, but the main one is that so many people are talking absolute rubbish about testing, and I want to set the record straight. Or, join in with more rubbish. Sure, I don't know everything, but I have a lot of experience and a lot of ideas.
Another reason is that I've found that I work on instinct a lot, yet get good results. So in addition to pointing out the rubbish and laughing at it, I hope to formulate my own approach into some sort of coherent framework that could be taught to others. It is easy to point out where something is wrong, it is much more difficult to suggest a solution.
My background, with anonymised company names: I started working in the Giant Fruit Company in Cork (Ireland) as a software tester in 1996, first on testing the porting of software to their range of laptops, and then testing localized software bundles. In early 1998, I moved to Dublin to work for the Coffee Cup Stain, my introduction to testing to spec, working on very large systems for GSM network management. In 2002, I went-all-Irish and emigrated to Finland, to work for A Finnish Telecoms Company (not the one you're thinking of) (nor the other one you're thinking of), which is where I finally began to have some input into how I was testing. Finally getting frustrated with the product I was testing, in May 2006 I moved to my current employer, The Logistics People, where I am the lead tester and (I like to think) shining example to all.
In addition: I am a rare being, a senior tester who also really likes executing tests. I have trouble taking things seriously, although inside, y'know, I'm like tortured. I move at my own pace, I'm easily bored, and I am far too busy outside of work.
I decided to start blogging about testing for a host of reasons, but the main one is that so many people are talking absolute rubbish about testing, and I want to set the record straight. Or, join in with more rubbish. Sure, I don't know everything, but I have a lot of experience and a lot of ideas.
Another reason is that I've found that I work on instinct a lot, yet get good results. So in addition to pointing out the rubbish and laughing at it, I hope to formulate my own approach into some sort of coherent framework that could be taught to others. It is easy to point out where something is wrong, it is much more difficult to suggest a solution.
My background, with anonymised company names: I started working in the Giant Fruit Company in Cork (Ireland) as a software tester in 1996, first on testing the porting of software to their range of laptops, and then testing localized software bundles. In early 1998, I moved to Dublin to work for the Coffee Cup Stain, my introduction to testing to spec, working on very large systems for GSM network management. In 2002, I went-all-Irish and emigrated to Finland, to work for A Finnish Telecoms Company (not the one you're thinking of) (nor the other one you're thinking of), which is where I finally began to have some input into how I was testing. Finally getting frustrated with the product I was testing, in May 2006 I moved to my current employer, The Logistics People, where I am the lead tester and (I like to think) shining example to all.
In addition: I am a rare being, a senior tester who also really likes executing tests. I have trouble taking things seriously, although inside, y'know, I'm like tortured. I move at my own pace, I'm easily bored, and I am far too busy outside of work.
Subscribe to:
Posts (Atom)