Collaborative Software Testing

Getting Started With Exploratory Testing - Part 4

Practice, Practice, Practice

To practice exploratory testing, you will find that you need to be able to generate test ideas, sometimes at a moment's notice. The sky (or more accurately your mind) is the limit to the kinds of tests you can implement and practice with. The more you think up new tests, and learn to analyze software from many different angles, the more thorough your testing and the resulting discoveries will be. Like anything else, the more you practice, the better you will get.

Test Idea Strategies
Here are some test strategies you can use to generate test ideas to practice exploratory testing. Practice testing, but also practice thinking about testing, applying strategies and doing analysis work.

Concrete Strategies
Sometimes as a tester you need to grab and use specific test ideas off the shelf to get started quickly. Do you have a source of ideas for specific strategies tests sequences, or test inputs? Some test teams have quick reference lists with specific test ideas for their applications under test for this purpose. If you are practicing exploratory testing, and find you need some new ideas quickly, here are some places to look.

Michael Hunter has a fabulous list of test ideas in his You Are Not Done Yet blog series.

Elisabeth Hendrickson's Test Heuristics Cheat Sheet describes "data type attacks & web tests".

Karen Johnson has some concrete test ideas on her blog as well. Check out Testing by the Numbers, Chars, Strings and Injections, and ; SQL injections- -' for some ideas.

Concrete test ideas can be incredibly useful. Recently I decided to grab some of my favorite testing numbers off the shelf and input them through a test interface I had just developed (written in Java using a library which allowed me to write commands in JUnit tests.) I found a bug within seconds thanks to a testing interface that allowed me to use more speed and accuracy in my test inputs, and my off-the-shelf testing numbers.

Abstract Strategies
Abstract strategies are all about testing ideas. These require more thinking than a quick off the shelf test, but the benefits are enormous. Applying these kinds of strategies to testing can result in very thorough, creative, and disciplined test strategies and execution. You can use these strategies to help design tests in the moment, or while doing test analysis or test planning. Some examples include test models and heuristics.

James Bach's Heuristic Risk-Based Test Strategy Model is a great place to start. In fact, James' methodology page is full of great ideas for testers. I touched on the subject of a benefit of using models in this blog post Speeding Up Observation with Models.

Thanks to the work of James, there are several testing mnemonics you can use, or better yet, use as inspiration to create your own.

• Here is the classic "San Francisco Depot" from James (great general purpose testing mnemonic)
• Michael Bolton spells out HICCUPPS (testing without a map)
• Mike Kelly spells out FCC CUTS VIDS (for product tours, which I also represented as a mind map)
• Scott Barber spells out CCD IS EERI (useful for performance testing)

I touched on the subject of learning testing heuristics here.

Other models include coverage models, failure mode analysis, McLuhan thinking, and many others.

Language Strategies
This is another abstract strategy, but language is so important I decided to give it its own category. Examples of strategies are to look at the meaning of words, how they are put together, and how they might be interpreted in different ways. I use dictionaries a lot to help my thinking, and they can be a great tool to expose vague terms. I also use grammar rules when I investigate a product and look for potential problem areas. I engage in almost a type of "software hermeneutics." The linguistics (study of language) field has a wealth of tools we can also use to help generate test ideas.

The requirements-based testing community have a good handle on testing written documentation. When I look at anything that is written down that has to do with the product, I carefully look at the wording. That includes in the software, in any supporting documentation, comments, tests, etc. and I look for potential ambiguities. For example, Bender RBT Inc, has a paper, the "The Ambiguity Review Process" which contains a section: "List of Words that Point to Potential Ambiguities".

Once you find an ambiguity, can you interpret the usage of the software in different ways and test according to different definitions? This is often a surprising source of new test ideas. I use language analysis a lot as a tester, and it is a powerful method for me to generate test ideas.

Sensing and Feeling Strategies
Using Emotions
This is can include situations where our senses are telling us something that we aren't immediately aware of. We feel a sense of discomfort, sometimes called "cognitive dissonance". Things appear to be functioning, but we feel conflicted, sometimes for reasons that aren't readily apparent . I've learned to use the "feeling that something is off" as a clue to start investigating that area much more heavily. It almost always pays off in the discovery of a good bug.

Michael Bolton describes this emotional side of testing very well:

"...When I feel one of those emotional reactions, I pause and ask "Why do I feel this way?" Often, it's because the application is violating some consistency heuristic--it's not working as it used to; it's not supporting our company's image; there's a claim, made in some document or meeting or conversation, that's being violated; the product is behaving in a way that's inconsistent with a comparable product; ..."

Update: Be sure to check out Michael's excellent presentation on this topic here. It might change the way you think about testing.

Using Your Senses
Have you ever pushed a machine to its limits when testing and noticed something physically change in your test environment? This is also a trigger for me to push harder in that direction, because a discovery is on the way.
Here are some examples:

• sluggish response when interacting with the UI, or an API
• interaction slowing down
• sense the machine labouring
• hear the fan kick in hard when running a test
• feel the device getting hot

An old rule of thumb when traveling is to use a map, but follow the terrain. Being aware of any strange feelings you have when testing (it feels off, let's investigate this a bit more), and being aware of changes to the test environment are great ways to explore the system.

George Dinwiddie reminded me of a good rule of thumb:

"As Bob Pease (an analog EE) used to say in Electronic Design and EDN magazines, 'If you notice anything funny, record the amount of funny.'"

As a tester, when I see something funny, my first instinct is to get the application to repeat the funny behavior. Then I observe whether follow-on testing creates even more funny behavior. In many cases it does, because bugs often tend to cluster.

Some of my most significant discoveries have come about because I paid attention to a weird feeling, investigated further, noticed something flicker out of the corner of my eye, or noticed that the log file on a server that I was tailing was writing at a strange rate, or that a certain action caused my machine to start laboring, etc. It's often the subtle things that are the biggest clues to overlooked problems in the system, so it's a good idea to practice your observation skills.

Oblique Strategies
These are ideas you can use to spur creative thought processes if you find yourself running out of ideas. Michael Bolton introduced me to these last year while we were working on a testing exercise. I was running out of ideas, and Michael sent me to this web site that generates oblique strategies. After a couple of tries, I ended up on a whole new idea set that jump-started my testing.

Wikipedia says this about Oblique Strategies:
"...is a set of published cards created by Brian Eno and Peter Schmidt. Now in its fifth edition, it was first published in 1975. Each card contains a phrase or cryptic remark which can be used to break a deadlock or dilemma situation."

When I was a kid, my family had a game where you would blindly open a dictionary, close your eyes, and put your finger down on a page. You would then open your eyes, look at the word closest to your finger, and then make up a short story involving that word. A variation involved doing this for two words, and then combining them to try to come up with something creative or funny. This is a similar kind of idea generating strategy to oblique strategies, but not as sophisticated. There are other, similar ways of getting oneself out of a rut.

With these sorts of tools you often need to be creative to apply your result to testing, but with a little bit of trial and error and creative thinking, you will be surprised at what you come up with.

Uncategorized, or Inexplicable Strategies
Sometimes, believe it or not, some of my friends dislike my need to systematize, categorize and neologize. They prefer to think that their testing ideas are the work of intuition, or are dropped to them by the testing muse, or the idea fairy. That's fine, and I can respect that they think about things differently. Furthermore, I frequently miss out on ideas and insights that they have. I've added this section to include the mysterious, the unexplained, or any other test strategy I might not be aware of, or be hard-pressed to categorize.

What works for you? Do you use a strategy for testing that I haven't touched on in this post? This list is in no way exhaustive, and it, like any other model, is imperfect. I'm sure my own model and strategies will change as I learn more, and get more feedback from colleagues, and you, the reader.

I hope this list of test strategies helps you as you practice exploratory testing. Test idea generation can be as creative as we permit ourselves to explore, and one of the wonderful things about software testing is that sometimes the weird idea combinations are the best ones. Furthermore, practice helps build the strength and agility of the most powerful testing tool we have at our disposal, the human mind.

Getting Started With Exploratory Testing - Part 3

Wean Yourself off Scripted Tests

Scripted tests, or test cases that are written in advance, contain procedural steps to execute, and usually have some sort of "expected results" at the end are very common. If you are a tester who is used to only using these kinds of test cases, transitioning to exploratory testing has special challenges.

One way to get started with exploratory testing is to change the way you implement scripted tests, with two goals in mind:

1. to get a feeling for exploratory testing
2. to be able to engage in testing without needing pre-recorded test scripts

Some testers can find it overwhelming and can almost feel a sense of panic when they consciously try out exploratory testing. Exploratory testing is actually a natural activity for us to do, but sometimes we are so conditioned to follow a script, and only test what we've been told to test, it can be a difficult transition to make at first. There are some useful things to remember:

• If you feel overwhelmed by possibilities, stop, gather your thoughts, then pick one simple top priority item to test.
• There is no right or wrong way of practicing exploratory testing; your own unique creative and idea generation methods are a source of strength.
• Minimal planning can go a long way. To start, set out a time period (an hour might be a good place to start), and an area of focus in an application.
• For your self-allotted time box of exploratory testing, try not to multitask. Stick to your area of focus, and resist the urge to check email, answer colleagues' questions, or test other applications. If you find areas to explore outside of your area of focus, quickly take note of them, and return to them after your time-boxed testing session.

One way to do this is to encourage yourself to add more variability when exercising a test script. Elisabeth Hendrickson talks a lot about adding variability to testing in her work. One way to consciously add variability is to change the program inputs that are recommended in the test script you are following. You can change them slightly, or dramatically, feel free to experiment with different types, or use some of your "old favorite" inputs, write a script to generate test data, or use something like PerlClip.

Another way to consciously add variability into your test script is to vary the steps. Here is an analogy. A friend of mine is an accomplished musician with strong technical skills, but struggles with improvisation. Here is an exercise we did recently:

1. I recommended that she take a simple music score (like Mary Had a Little Lamb), and try to play it note for note. I wanted her to play a rendition that she felt was as close to the way the author intended the sheet music to be interpreted.
2. Next, I asked her to add ornamentation to her playing. (This is subtle variation, or a slightly unique style that comes out when playing an instrument, which tends to be unique from musician to musician.)
3. Once she had done that, I asked her to play the song again, but to ignore some of the bars in the sheet music. I asked her to anticipate ending up on an important, signature part of the song so it was still recognizable, but encouraged her to do whatever came to mind to get there. It could be silence with counting and waiting; it could be one extended note; it could involve running up and down an appropriate scale; trying to play what they thought should be there from memory, or what was going on in her head.
4. With that out of the way, I told her to throw caution to the wind and play with complete freedom. I asked her to play for roughly the same amount of time as the song might take, but to play whatever came to mind.

I asked her to record and send me her versions of the song at each step. The results were surprising and satisfying. In step 1, the song sounded as you would expect, but was a bit wooden, or forced, like a computer had played it. In step 2, the performance came alive. In step 3, there were flashes of brilliance and the odd spot where execution went off the rails. Part 4 was a huge surprise. My friend had been listening to progressive heavy metal earlier in the day, and the piece she came up with while improvising didn't sound like Mary Had a Little Lamb at all. It sounded like part of a Dream Theater song. The notes were there, the scales were there, in fact, most of the framework of what Mary Had a Little Lamb is composed of was there, but it wasn't recognizable. However, it was incredibly effective as performed music. It was interesting, entertaining, creative, and it flowed with an effortlessness that resonates and connects with a listener. "Flow" is an important concept both for musical improvisation, and for exploratory testing.

With my musician friend, my goals were to get her to feel "flow" when playing her instrument, and to meld her thought processes with her physical manipulation of the instrument. Most importantly, I wanted her to let go of her inhibition when playing, to not worry about breaking the rules, and to just be free and have fun while achieving a goal. "Flow" is not only important in creative endeavors, it applies to any kind of human/computer interaction. This article is a good overview of the concept from a gaming perspective.

Do you have enough information to apply this kind of thinking to your testing work? If you're still stuck, here are some things to consider:

Contrary to some perpetuated testing folklore, you do not need "Expected Results" to be written down prior to testing. James Bach says:

"The expected result thing is interesting because we have a million expectations, mostly unconscious and emergent. To suggest that there are only one or two expected results is absurd. What I say is not that you need an expected result for a test. What you need is a reasonable oracle. This is a broader idea than expected result. By the way, your oracle is not necessarily needed before the test is executed. It can come at any time, even a week later."

Note: An "oracle" is defined as: "...a principle or mechanism by which we recognize a problem." Also notice that in my example above, the "expected results" are not written down in the sheet music, they are implied expectations of the performer and the listeners. Not having them written down is no more an impediment to testing than it is not an impediment to performing music.

You do not need formal requirements, or a complete specification to test. As a software user and tester, you have what James Bach calls "reasonable expectations". Michael Bolton has a nice article that addresses this issue: "Testing Without a Map". You can use some of your scripted test cases as a place to start, but one goal for your exploratory testing should be doing something like what Michael describes here.

To start weaning yourself off of scripted tests, try to repeat what my musician friend did above, but with test cases you are familiar with. If you get stuck trying to transition away from a scripted test, here are some things to try.

• Execute a test script, but stop part way though, and interact with the application at whatever point you are at, then come back to the test script. Does following the script feel different? Take note of that feeling, and try to replicate the feeling you get with exploratory testing more often in your testing execution, and when thinking about testing.
• Stop part way through the script, and engage in another activity. Stare out the window, take a short walk, or read a testing article, and then come back to your test script. Instead of following the rest of the test script steps, focus on the first thing you see in the application, and interact with that component.
• Memorize the test script, re-run it, and then get creative in your execution of it. Can you change it so much that it still meets the intent or goal of the script, but the original writer may not recognize it?
• Try to be as observant as possible. Can you see anything different that you might not have noticed before? Stop and take notice of what the application is doing. Does anything feel different or suspicious to you? Is there a blink, or an oddity in the way a screen loads? If something feels funny, this is a good sign to investigate that thing that seems a bit off. Embrace those feelings when they come, that is often a sign that you are almost subconsciously observing something different.
• Research other ways of interacting and implementing tests, and pre-plan a completely new strategy for testing that application. Can you use a different testing tool? Execute those tests through a different interface than the user interface? Once you have a new plan of attack, follow the intention of the script, but not the steps of the script using a different testing strategy or technique.
• Focus on observing slight details when testing, especially ones you didn't see before, and capitalize on exploring them. Practice developing your observation skills away from the computer, and apply your learnings to testing.
• For something completely different, try testing James Lyndsay's black box test machines. There's nothing quite like testing something that has no written documentation at all to go on.

With time, you should find that you no longer need pre-scripted tests at all to begin testing. All you will need is software to test, and your testing ideas and thinking skills. If you can learn to alternate between both scripted and exploratory testing, your testing and your thinking about testing will change. I hope this will help testing become more interesting for you, particularly if you feel locked in scripted testing mode. I also hope that if you practice and think about testing, your confidence in your abilities as a tester will grow.

Getting Started with Exploratory Testing - Part 2

Apply Improvisation to Testing

How do I get started with exploratory testing? Here are is an idea to help you get started:

Study and practice improvisation, and apply your learnings to your own testing work.

Wikipedia describes improvisation as:

"...the practice of acting and reacting, of making and creating, in the moment and in response to the stimulus of ones immediate environment. This can result in the invention of new thought patterns, new practices, new structures or symbols and/or new ways to act. This invention cycle occurs most effectively when the practitioner has a thorough intuitive or technical understanding of the necessary skills and concerns within the improvised domain."

I first consciously studied and practiced improvisation (improv) in middle school. In drama class, we were encouraged to study and practice improv to develop our acting skills. I enjoyed being in the moment, and working with whatever props, cues or events I had at the time to work with. I realized quite quickly that it is a lot harder than it looks; reading from a script and being directed on everything you do in a performance is very different. It's easy to get over confident and then freeze in a performance if you are unprepared.

There are many skills that actors can develop to do improv well. Some of my friends practiced different techniques with their faces, posture, body language and voices so they could be even more versatile. Some could change the entire mood they wanted to convey with a few imperceptible changes in their demeanor. That requires practice and skill, but good actors are able to employ these kinds of tools almost effortlessly once they are mastered.

I then studied improvisation in music. After high school, I joined a band that needed a saxophonist. They liked to do extended jams, and it wasn't easy to keep up at first. I was well practiced reading sheet music and had my parts all polished up, but I quickly fell off the rails when the band leader would lead improvised sessions. I'd miss a key change and do something dumb like playing something minor-dominant when we had changed to a major key, or get my rhythm mixed up because the drummer had changed timing while I was in the middle of a solo. After I practiced like mad, and learned a couple of patterns, I could cope a lot better with these free sessions, but I never had the chops to play in free jazz bands like some of my friends have.

Some people have said: "Oh, Jazz? Is that the music where they just make it up as they go along?" This is actually incredibly difficult to do. My friends who have played in jazz bands are technical monsters on their instruments. They have very strong knowledge of music theory, and can play their instruments with great precision and speed. I on the other hand, have a much more narrow grasp as a musician, but I can still do some improvisation. I could spend the rest of my life just trying to master guitar.

Like most things in life, improvisation can be done poorly, or it can appear effortless and mesmerizing when done by a skilled practitioner. It can be done effectively by people with various degrees of skill. I have musician friends who are musical and very strong technically on their instruments freeze up without sheet music. I also have musician friends who know very little music theory who can do very well at improvisation. I also have testing friends who have fabulous improvisation skills, and others who do it naturally and seemingly unconsciously, but effectively.

Improvisation is about being in the moment, and dealing with what you observe, think, feel and react to. As testers, we can be in the moment as we interactively test software, and we can be as free and unconstrained as other improvisers. To do it very well requires skill, but we can be effective and grow our testing skills if we learn and practice. We just need to be aware of the software we are testing, and learn to keenly observe what it is doing, and be able to react and create new tests as we interact with it. Like acting, music, or other improvisational activities, you will find you end up in interesting and rewarding situations.

*See Dictionary.com for definitions of improvise.