Collaborative Software Testing

Getting Started With Exploratory Testing - Part 3

Wean Yourself off Scripted Tests

Scripted tests, or test cases that are written in advance, contain procedural steps to execute, and usually have some sort of "expected results" at the end are very common. If you are a tester who is used to only using these kinds of test cases, transitioning to exploratory testing has special challenges.

One way to get started with exploratory testing is to change the way you implement scripted tests, with two goals in mind:

1. to get a feeling for exploratory testing
2. to be able to engage in testing without needing pre-recorded test scripts

Some testers can find it overwhelming and can almost feel a sense of panic when they consciously try out exploratory testing. Exploratory testing is actually a natural activity for us to do, but sometimes we are so conditioned to follow a script, and only test what we've been told to test, it can be a difficult transition to make at first. There are some useful things to remember:

• If you feel overwhelmed by possibilities, stop, gather your thoughts, then pick one simple top priority item to test.
• There is no right or wrong way of practicing exploratory testing; your own unique creative and idea generation methods are a source of strength.
• Minimal planning can go a long way. To start, set out a time period (an hour might be a good place to start), and an area of focus in an application.
• For your self-allotted time box of exploratory testing, try not to multitask. Stick to your area of focus, and resist the urge to check email, answer colleagues' questions, or test other applications. If you find areas to explore outside of your area of focus, quickly take note of them, and return to them after your time-boxed testing session.

One way to do this is to encourage yourself to add more variability when exercising a test script. Elisabeth Hendrickson talks a lot about adding variability to testing in her work. One way to consciously add variability is to change the program inputs that are recommended in the test script you are following. You can change them slightly, or dramatically, feel free to experiment with different types, or use some of your "old favorite" inputs, write a script to generate test data, or use something like PerlClip.

Another way to consciously add variability into your test script is to vary the steps. Here is an analogy. A friend of mine is an accomplished musician with strong technical skills, but struggles with improvisation. Here is an exercise we did recently:

1. I recommended that she take a simple music score (like Mary Had a Little Lamb), and try to play it note for note. I wanted her to play a rendition that she felt was as close to the way the author intended the sheet music to be interpreted.
2. Next, I asked her to add ornamentation to her playing. (This is subtle variation, or a slightly unique style that comes out when playing an instrument, which tends to be unique from musician to musician.)
3. Once she had done that, I asked her to play the song again, but to ignore some of the bars in the sheet music. I asked her to anticipate ending up on an important, signature part of the song so it was still recognizable, but encouraged her to do whatever came to mind to get there. It could be silence with counting and waiting; it could be one extended note; it could involve running up and down an appropriate scale; trying to play what they thought should be there from memory, or what was going on in her head.
4. With that out of the way, I told her to throw caution to the wind and play with complete freedom. I asked her to play for roughly the same amount of time as the song might take, but to play whatever came to mind.

I asked her to record and send me her versions of the song at each step. The results were surprising and satisfying. In step 1, the song sounded as you would expect, but was a bit wooden, or forced, like a computer had played it. In step 2, the performance came alive. In step 3, there were flashes of brilliance and the odd spot where execution went off the rails. Part 4 was a huge surprise. My friend had been listening to progressive heavy metal earlier in the day, and the piece she came up with while improvising didn't sound like Mary Had a Little Lamb at all. It sounded like part of a Dream Theater song. The notes were there, the scales were there, in fact, most of the framework of what Mary Had a Little Lamb is composed of was there, but it wasn't recognizable. However, it was incredibly effective as performed music. It was interesting, entertaining, creative, and it flowed with an effortlessness that resonates and connects with a listener. "Flow" is an important concept both for musical improvisation, and for exploratory testing.

With my musician friend, my goals were to get her to feel "flow" when playing her instrument, and to meld her thought processes with her physical manipulation of the instrument. Most importantly, I wanted her to let go of her inhibition when playing, to not worry about breaking the rules, and to just be free and have fun while achieving a goal. "Flow" is not only important in creative endeavors, it applies to any kind of human/computer interaction. This article is a good overview of the concept from a gaming perspective.

Do you have enough information to apply this kind of thinking to your testing work? If you're still stuck, here are some things to consider:

Contrary to some perpetuated testing folklore, you do not need "Expected Results" to be written down prior to testing. James Bach says:

"The expected result thing is interesting because we have a million expectations, mostly unconscious and emergent. To suggest that there are only one or two expected results is absurd. What I say is not that you need an expected result for a test. What you need is a reasonable oracle. This is a broader idea than expected result. By the way, your oracle is not necessarily needed before the test is executed. It can come at any time, even a week later."

Note: An "oracle" is defined as: "...a principle or mechanism by which we recognize a problem." Also notice that in my example above, the "expected results" are not written down in the sheet music, they are implied expectations of the performer and the listeners. Not having them written down is no more an impediment to testing than it is not an impediment to performing music.

You do not need formal requirements, or a complete specification to test. As a software user and tester, you have what James Bach calls "reasonable expectations". Michael Bolton has a nice article that addresses this issue: "Testing Without a Map". You can use some of your scripted test cases as a place to start, but one goal for your exploratory testing should be doing something like what Michael describes here.

To start weaning yourself off of scripted tests, try to repeat what my musician friend did above, but with test cases you are familiar with. If you get stuck trying to transition away from a scripted test, here are some things to try.

• Execute a test script, but stop part way though, and interact with the application at whatever point you are at, then come back to the test script. Does following the script feel different? Take note of that feeling, and try to replicate the feeling you get with exploratory testing more often in your testing execution, and when thinking about testing.
• Stop part way through the script, and engage in another activity. Stare out the window, take a short walk, or read a testing article, and then come back to your test script. Instead of following the rest of the test script steps, focus on the first thing you see in the application, and interact with that component.
• Memorize the test script, re-run it, and then get creative in your execution of it. Can you change it so much that it still meets the intent or goal of the script, but the original writer may not recognize it?
• Try to be as observant as possible. Can you see anything different that you might not have noticed before? Stop and take notice of what the application is doing. Does anything feel different or suspicious to you? Is there a blink, or an oddity in the way a screen loads? If something feels funny, this is a good sign to investigate that thing that seems a bit off. Embrace those feelings when they come, that is often a sign that you are almost subconsciously observing something different.
• Research other ways of interacting and implementing tests, and pre-plan a completely new strategy for testing that application. Can you use a different testing tool? Execute those tests through a different interface than the user interface? Once you have a new plan of attack, follow the intention of the script, but not the steps of the script using a different testing strategy or technique.
• Focus on observing slight details when testing, especially ones you didn't see before, and capitalize on exploring them. Practice developing your observation skills away from the computer, and apply your learnings to testing.
• For something completely different, try testing James Lyndsay's black box test machines. There's nothing quite like testing something that has no written documentation at all to go on.

With time, you should find that you no longer need pre-scripted tests at all to begin testing. All you will need is software to test, and your testing ideas and thinking skills. If you can learn to alternate between both scripted and exploratory testing, your testing and your thinking about testing will change. I hope this will help testing become more interesting for you, particularly if you feel locked in scripted testing mode. I also hope that if you practice and think about testing, your confidence in your abilities as a tester will grow.

Posted by jonathankohl at May 25, 2007 03:19 PM