Category Archives: mnemonics

Getting Started With Exploratory Testing – Part 4

Practice, Practice, Practice

To practice exploratory testing, you will find that you need to be able to generate test ideas, sometimes at a moment’s notice. The sky (or more accurately your mind) is the limit to the kinds of tests you can implement and practice with. The more you think up new tests, and learn to analyze software from many different angles, the more thorough your testing and the resulting discoveries will be. Like anything else, the more you practice, the better you will get.

Test Idea Strategies

Here are some test strategies you can use to generate test ideas to practice exploratory testing. Practice testing, but also practice thinking about testing, applying strategies and doing analysis work.

Concrete Strategies

Sometimes as a tester you need to grab and use specific test ideas off the shelf to get started quickly. Do you have a source of ideas for specific strategies tests sequences, or test inputs? Some test teams have quick reference lists with specific test ideas for their applications under test for this purpose. If you are practicing exploratory testing, and find you need some new ideas quickly, here are some places to look.
Michael Hunter has a fabulous list of test ideas in his You Are Not Done Yet blog series.

Elisabeth Hendrickson’s Test Heuristics Cheat Sheet describes “data type attacks & web tests”.

Karen Johnson has some concrete test ideas on her blog as well. Check out Testing by the Numbers, Chars, Strings and Injections, and ; SQL injections- -‘ for some ideas.

Concrete test ideas can be incredibly useful. Recently I decided to grab some of my favorite testing numbers off the shelf and input them through a test interface I had just developed (written in Java using a library which allowed me to write commands in JUnit tests.) I found a bug within seconds thanks to a testing interface that allowed me to use more speed and accuracy in my test inputs, and my off-the-shelf testing numbers.

Abstract Strategies

Abstract strategies are all about testing ideas. These require more thinking than a quick off the shelf test, but the benefits are enormous. Applying these kinds of strategies to testing can result in very thorough, creative, and disciplined test strategies and execution. You can use these strategies to help design tests in the moment, or while doing test analysis or test planning. Some examples include test models and heuristics.

James Bach’s Heuristic Risk-Based Test Strategy Model is a great place to start. In fact, James’ methodology page is full of great ideas for testers. I touched on the subject of a benefit of using models in this blog post Speeding Up Observation with Models.

Thanks to the work of James, there are several testing mnemonics you can use, or better yet, use as inspiration to create your own.

I touched on the subject of learning testing heuristics here.

Other models include coverage models, failure mode analysis, McLuhan thinking, and many others.

Language Strategies

This is another abstract strategy, but language is so important I decided to give it its own category. Examples of strategies are to look at the meaning of words, how they are put together, and how they might be interpreted in different ways. I use dictionaries a lot to help my thinking, and they can be a great tool to expose vague terms. I also use grammar rules when I investigate a product and look for potential problem areas. I engage in almost a type of “software hermeneutics.” The linguistics (study of language) field has a wealth of tools we can also use to help generate test ideas.

The requirements-based testing community have a good handle on testing written documentation. When I look at anything that is written down that has to do with the product, I carefully look at the wording. That includes in the software, in any supporting documentation, comments, tests, etc. and I look for potential ambiguities. For example, Bender RBT Inc, has a paper, the “The Ambiguity Review Process” which contains a section: “List of Words that Point to Potential Ambiguities“.

Once you find an ambiguity, can you interpret the usage of the software in different ways and test according to different definitions? This is often a surprising source of new test ideas. I use language analysis a lot as a tester, and it is a powerful method for me to generate test ideas.

Sensing and Feeling Strategies

Using Emotions

This is can include situations where our senses are telling us something that we aren’t immediately aware of. We feel a sense of discomfort, sometimes called “cognitive dissonance”. Things appear to be functioning, but we feel conflicted, sometimes for reasons that aren’t readily apparent . I’ve learned to use the “feeling that something is off” as a clue to start investigating that area much more heavily. It almost always pays off in the discovery of a good bug.

Michael Bolton describes this emotional side of testing very well:

…When I feel one of those emotional reactions, I pause and ask “Why do I feel this way?” Often, it’s because the application is violating some consistency heuristic–it’s not working as it used to; it’s not supporting our company’s image; there’s a claim, made in some document or meeting or conversation, that’s being violated; the product is behaving in a way that’s inconsistent with a comparable product; …

Update: Be sure to check out Michael’s excellent presentation on this topic here. It might change the way you think about testing.

Using Your Senses

Have you ever pushed a machine to its limits when testing and noticed something physically change in your test environment? This is also a trigger for me to push harder in that direction, because a discovery is on the way.
Here are some examples:

  • sluggish response when interacting with the UI, or an API
  • interaction slowing down
  • sense the machine labouring
  • hear the fan kick in hard when running a test
  • feel the device getting hot

An old rule of thumb when traveling is to use a map, but follow the terrain. Being aware of any strange feelings you have when testing (it feels off, let’s investigate this a bit more), and being aware of changes to the test environment are great ways to explore the system.

George Dinwiddie reminded me of a good rule of thumb:

As Bob Pease (an analog EE) used to say in Electronic Design and EDN magazines, ‘If you notice anything funny, record the amount of funny.’

As a tester, when I see something funny, my first instinct is to get the application to repeat the funny behavior. Then I observe whether follow-on testing creates even more funny behavior. In many cases it does, because bugs often tend to cluster.

Some of my most significant discoveries have come about because I paid attention to a weird feeling, investigated further, noticed something flicker out of the corner of my eye, or noticed that the log file on a server that I was tailing was writing at a strange rate, or that a certain action caused my machine to start laboring, etc. It’s often the subtle things that are the biggest clues to overlooked problems in the system, so it’s a good idea to practice your observation skills.

Oblique Strategies

These are ideas you can use to spur creative thought processes if you find yourself running out of ideas. Michael Bolton introduced me to these last year while we were working on a testing exercise. I was running out of ideas, and Michael sent me to this web site that generates oblique strategies. After a couple of tries, I ended up on a whole new idea set that jump-started my testing.

Wikipedia says this about Oblique Strategies: “…is a set of published cards created by Brian Eno and Peter Schmidt. Now in its fifth edition, it was first published in 1975. Each card contains a phrase or cryptic remark which can be used to break a deadlock or dilemma situation.”

When I was a kid, my family had a game where you would blindly open a dictionary, close your eyes, and put your finger down on a page. You would then open your eyes, look at the word closest to your finger, and then make up a short story involving that word. A variation involved doing this for two words, and then combining them to try to come up with something creative or funny. This is a similar kind of idea generating strategy to oblique strategies, but not as sophisticated. There are other, similar ways of getting oneself out of a rut.

With these sorts of tools you often need to be creative to apply your result to testing, but with a little bit of trial and error and creative thinking, you will be surprised at what you come up with.

Uncategorized, or Inexplicable Strategies

Sometimes, believe it or not, some of my friends dislike my need to systematize, categorize and neologize. They prefer to think that their testing ideas are the work of intuition, or are dropped to them by the testing muse, or the idea fairy. That’s fine, and I can respect that they think about things differently. Furthermore, I frequently miss out on ideas and insights that they have. I’ve added this section to include the mysterious, the unexplained, or any other test strategy I might not be aware of, or be hard-pressed to categorize.

What works for you? Do you use a strategy for testing that I haven’t touched on in this post? This list is in no way exhaustive, and it, like any other model, is imperfect. I’m sure my own model and strategies will change as I learn more, and get more feedback from colleagues, and you, the reader.

I hope this list of test strategies helps you as you practice exploratory testing. Test idea generation can be as creative as we permit ourselves to explore, and one of the wonderful things about software testing is that sometimes the weird idea combinations are the best ones. Furthermore, practice helps build the strength and agility of the most powerful testing tool we have at our disposal, the human mind.

Getting Started with Exploratory Testing – Part 1

Apply Investigation to Testing

How do I get started with exploratory testing? Here is an idea to help you get started:

Research investigative disciplines, and apply your learnings to your own testing work.

Software testing is investigative. There are investigative disciplines all around us that we can learn from. The obvious ones are the ones with “investigator” in their title – detectives, fire investigators, journalists, and others. However, other disciplines are also investigative. I have a friend who is a pilot, and he constantly investigates the complexities around operating an aircraft while he is flying. I have a friend who is a nurse, who needs to constantly investigate symptoms and react in first-line response when dealing with patients at a hospital. I have a friend who is a doctor, and her work is investigative, whether it is diagnosing a patient with a cold, looking over test results or performing surgery. I have a friend who is a firefigher, and he uses investigative skills to secure a scene so the firefighters can safely do their work. He also does vehicle extrications at car crashes, and goes out to medical calls.

I ask them about their work, the tools they use, and listen to their stories. All of the people listed above use some similar thinking tools. For example, they all use mnemonics. Why? Each of the careers I mention above deal with mission critical situations, and they want to have a repeatable, thorough thinking process. Mnemonics are memory aids, and help investigators be consistent and thorough as they observe and think about problems they are facing. Mnemonics are also a type of model that help speed up observation. Exploratory testers use mnemonics as well. There are other models and tools that they use that are fascinating to learn more about.

I also talk to others who use investigative tools in their work when I can. I’ve talked to historians, auto mechanics, forensic accountants, and my dental hygenist. Another source of information for exploratory testers are books on investigative fields. Testing Consultant Rob Sabourin is a big Sherlock Holmes fan, and Mike Kelly recommends the book “Conned Again Watson” which deals with probability, statistics, logic, etc.

Michael Bolton recently recommended the book: “How Doctors Think”, which is another reference for studying an investigative discipline. Ben Simo has a great write up of his impressions of this book on his blog. When I talk to my health care worker friends, I notice a lot of similarities to my work, and I learn something to apply to my testing thinking.

Once you have some new ideas, practice some of them during your daily testing activities. You will probably notice that you start thinking about testing differently, the more you apply investigative techniques to your own work.

*See Dictionary.com for definitions of investigate.

Modeling Test Heuristics

Mike Kelly has created an effective software testing mnemonic based around James Bach’s software touring heuristic. It is now burned indelibly on my brain: FCC CUTS VIDS. I think Mike hit a home run with this one – I use it a lot myself. The mnemonic makes more sense when you review Mike’s excellent explanation, and when he spells out the mnemonic into a list:

Feature tour
Complexity tour
Claims tour

Configuration tour
User tour
Testability tour
Scenario tour

Variability tour
Interoperability tour
Data tour
Structure tour

Like all good mnemonics, it is easy to memorize because it evokes familiarity, imagery, and it has a nice cadence. I should qualify that. It is easy for /me/ to memorize, and of course, as the creator, easy for Mike to memorize. I relate to words and lists, and a nice cadence helps me repeat it silently to myself, in English, the language I tend to think in. My friend Steve, one of the smartest people I know, does not have as much self-talk going on in his brain. He doesn’t relate much to lists and words, he relates to images and colors much more than I do. He thinks in pictures more than words. If I were teaching Steve the touring heuristic, and hoped he’d memorize it, I’d take a different tack.

To explore Mike’s test heuristic in an alternate form, I created the following image, in the form of a mind map, as an example:

This image is a model my friends who think more in terms of images and colors might be more comfortable with than a list.
As I look at it, I see some cool side effects. As a mind map, it looks unfinished. This is partly by design; I didn’t expand any of the idea points as you normally would, and there is a blank spot at the bottom right hand corner, an area that our eyes are naturally drawn to as we scan an image.

Contrast the image with the list above. In the list, I get a sense of completion – each letter of the mnemonic is spelled out. When I get to the final “S”, standing for “Structure Tour”, I get a sense of satisfaction. The spelling out of the mnemonic is complete. When I look at the image, I am struck by how unfinished it looks. I want to draw it on a whiteboard and expand each idea bubble. I want to fill in that blank spot at the bottom right-hand corner. I want to explore the heuristic further, and I want my image to be symmetrical so I can easily remember the details by the shape and colors in the model.

My next urge is to re-create the mind map using only images instead of text. That would be an interesting experiment. Next, I want to convert the list into a tree diagram, with all the relevant testing ideas I can fit under each item. A tree diagram is also an effective way to quickly spot areas that need to be expanded. This appeals to my pragmatic testing approach – more analysis can help me be that much more thorough the next time I use the mnemonic when testing.

Different models help us look at problems differently, and they will reveal or obscure different information and ideas. As my father taught me years ago, involving all your senses when learning is effective. I constantly analyze my own models, and try to improve on my technique. If a model is getting stale, all I need to do is change the medium, and something extraordinary usually clicks in my brain as a result.

What do you see in the image and list above? Do they evoke different reactions in your mind? How would you model Mike’s mnemonic? What tools would you use, and what would you learn about testing and thinking by employing them? How many of you would write a song to accompany it? There’s only one way to find out. Model away.

*Thanks to Sherry Heinze for reawakening my interest in mind maps.

Learning Testing Heuristics

When I was in junior high school, one of my tasks was to learn metric conversions. My science teacher at the time took this task seriously. To help us remember the kilo, hecto, deca, deci, centi, milli prefixes to meter, liter and gram, he had us memorize a mnemonic. The mnemonic looked like this: KHDMDCM, and there was a sort of song that went through it: “King Henry Danced Merrily…something something something”. Trouble was, I barely had the mnemonic memorized in time for a pop quiz, and I couldn’t remember what each letter stood for. I ended up only being able to write the King Henry song out on my test paper and handing it in. I couldn’t do the required conversions such as decimeters to meters, or kilometers to centimeters. My teacher gave me a cold look when he handed the exams back, and admonished me to start trying. Trouble was, I was trying. I just couldn’t get that stupid mnemonic to make sense in my brain.

At home that night, I sang the King Henry song over and over, repeating what each letter stood for. It was hopeless. I had to constantly look at my cue cards.

My older sister happened along and asked: “What on EARTH are you doing?”

I explained.

“Don’t do it that way,” she said, and quickly sketched out a diagram.

She wrote out all the measurement prefixes from largest to smallest, with “meter, liter, gram” in the center, and drew an arrow going up to the left on one side, and down to the right on the other. I stared at the diagram for a few seconds, and she had me reproduce it cold. Within minutes, I had the image memorized. She then taught me how to use it for conversion. It was a snap. On my metric conversion exam later that week, I ploughed through the questions with ease. “Forget King Henry,” was my thought, “this works way better.” And, forget him I did.

However, over twenty years later, I can still scratch out that conversion diagram and convert deciliters to decaliters, or kilograms to grams on the back of a napkin with relative ease.

“Different people learn differently” my father once told me. His first career was a school teacher, and over twenty five years he taught hundreds of different students. He was surprised by little, and was quite pragmatic. If colors helped you learn and remember, use colors. If mnemonics and little songs worked, try them. If images worked, go with the images. Try to involve most of your senses if you want to remember something, he said. Furthermore, if the system you are trying to use doesn’t work for you, make sure you clearly understand the concept, and then create your own. (These didactic sessions usually came from the depths of a newspaper, my Dad gazing at me through thick rimmed glasses with CBC Radio blasting from a radio in the background.)

When I started working with James Bach, he talked about using mnemonics for testing. I was familiar with, and had used most of his published mnemonics, but I had never memorized them. I used them more for analysis, or once in a while I’d scrawl one out on an index card and try to use it when testing. They seemed much more suited to analysis and planning though, and I was able to use them to generate extremely effective test strategies and plans.

Then one day I saw James at work, testing software. He was thinking out loud, using test heuristics, rattling off testing mnemonics, and simultaneously designing and executing tests under each letter of a mnemonic in real time. He would pause periodically to point out his findings or jot down notes. I’d never seen anything like it, and I knew right then I had to master it. I needed to memorize those mnemonics, so I too could quickly slice through a testing problem (any testing problem) with thorough ease. However, the spectre of King Henry loomed above me, reminding me of my early mnemonic memorization failure.

I had learned other mnemonics that were quite useful and effective, but memorizing mnemonics was a bit of a blind spot for me. Who can forget SOHCAHTOA, an effective way to remember how to calculate the sine, cosine and tangent of an angle? I remembered memorizing that mnemonic and using it to great effect. “Soak-a-toe-ah” I repeated in a unique cadence, and imagined a big toe in a wash basin. The imagery and the rhythmic cadence of “soak-a-toe-ah” helped me remember the mnemonic, and practice helped me remember the application. I could memorize testing mnemonics just as easily, and banish King Henry from my memorization nightmares.

James peered down at me over the rim of his glasses: “You need to memorize testing mnemonics.”

His tone was casual, but firm. To him it was like telling me I needed to learn to cross the street. Obvious. I complained about how I often found it hard to memorize them, and often used other kinds of models.

“That’s because you are memorizing someone else’s. You need to create your own. If they are yours, you’ll remember them.”

I was faithful to that advice, and it has served me well. I’ve added mnemonic memorization and creation to my testing thinking repertoire. I use mnemonics along with diagrams, images, songs, keywords, and guide words. When I teach testing techniques, I find, as my Father did many years ago, that some thinking models appeal to different people in different ways than others. Furthermore, using different thinking models with test heuristics helps me analyze them, enhance them, and learn more about my own work both during analysis and planning, and during execution.

EDIT Update: I SLICED UP FUN is a popular mnemonic I created. I developed it first for myself, then shared with my team, then with everyone else. It has become quite popular on mobile software development projects the world over. Are you curious about mnemonics? If so, imagine me peering over my glasses at you.

Speeding Up Observation with Models

You just generated four possible ways to solve the problem in about thirty seconds!

One of my brilliant programmer friends had just posed one of those useless interviewing puzzles that Microsoft made famous. We were discussing that the only thing they test is a candidate’s ability to solve those sorts of problems, and really tell you nothing about their ability to do software development tasks. He was frustrated by a recent interviewer who had merely posed a puzzle, wrinkled up their nose at his attempt to solve it in the interview, and had then lectured him on the “right” answer. To illustrate, he had posed the question to me.

Fortunately, it was a type of puzzle that I could easily apply a model against. Also in my favor, I had spent a recent evening teaching a software testing student that very model. I smiled inwardly, and with nonchalance outlined four possible ways of attacking the problem. He stared at me almost dumb-founded for a moment, and then I had to let him off the hook. I laughed as he stared at me in wonder. “There’s a trick to this” I told him. I revealed the model, which is the software testing mnemonic HICCUPPS. I suddenly had a brilliant programmer with a much stronger intellect than mine very interested in software testing. “Teach it to me!” he chortled, with relief and delight.

We both agreed that the “right” answer the interviewer posed was a complete joke, and he was justified in concluding that this was not a company he wanted to work for. That company lost out on hiring one of the best problem-solvers I know.

“Teach me more about these testing mnemonics!” he admonished as we parted ways that day.

* * *

How did you generate so much thorough analysis so quickly? I’ve been here for three weeks, and you’ve been here for three days!

I was doing analysis work with a friend of mine, who like my friend above, is also a brilliant programmer. When I was still in short pants, he was creating artificial intelligence programs in Lisp.

“Part of the reason is that I have your three weeks of work to draw on.” I replied. “Of course, but you have touched on areas I hadn’t thought about at all. I would have thought of them eventually, but how did you think of them so quickly?” he said with a questioning grin. “I use models”, I replied, and brought out a testing problem to demonstrate my thinking. He grasped the object I use in that problem for a second, then looked at me, handed it back and said: “Oh, I see how you do it. Raw observation is very slow. Your models help frame your thinking and speed up observation.” He didn’t even try to solve the problem, and in his characteristic way, rapidly understood a lesson that had taken me much longer to learn.

“Good work” he said with a chuckle as he got up to leave for home. My speech on the wonders of James Bach’s San Francisco Depot (SFDPOT) mnemonic would have to wait for another day.

* * *
Models are useful for generating ideas when solving problems. They aren’t fool-proof though. I am careful to not ignore, but to absorb and test my first instinctive thoughts when solving a problem. Often, that intuition helps, but sometimes I discard it, as careful scrutiny denies the plausibility of the initial thought impulse. Intuition can be both powerful and perilous. There’s nothing wrong with intuition, as long as you can make your conclusions defensible.

It doesn’t hurt to complement intuitive thought with a model, and I find using both helps kick start my thinking processes. Sometimes, the most lucid and effective ideas sprout out of nowhere from a sea of wonder and half-baked thoughts. They frequently pop into my head as I am using a model, and they often turn out to be unrelated to my current train of thought, but incredibly useful. I don’t ignore them merely because I can’t explain them. Instead, I try to adapt my models to increase the chances of them occurring.

Models also add consistency when solving problems. Instead of a formulaic, step-by-step, “one size fits all” approach to solve a problem, they can be used to consistently remind our brains to look into diverse areas for clues and impulses. This is much more powerful than a rote process that may, or as in most cases, may not work. It greatly increases our chances of coming up with a unique solution for a unique problem. I find that after a considerable degree of rumination, I have an “aha” moment, but using a model to complement my natural thinking usually brings me to an “aha – why didn’t I think of that before?!!” thought much more quickly.

Models also help build confidence. If I am posed with a difficult problem, and I have no idea where to start, I can nervously waver and pause, or I can jump in with confidence and use a model. It doesn’t matter if it’s the wrong one, the thinking drives out the fear and uncertainty and leads me in a direction towards solving the problem. Fear steals energy, while models give me a place to start in the face of uncertainty. I constantly learn from experience, and develop new models when I learn a new lesson. Variety of models makes the problem-solving space much richer, and less intimidating.

A good friend of mine is a firefighter, and he also uses mnemonics in his work. His son had recently taken an emergency services technician course, and he was drilling his knowledge one evening. They both have been taught mnemonics to help frame their thinking in emergency situations. In fact, they and others have told me that attempts at scripting this kind of problem solving ends up in farcical situations, or worse, loss of life.

That sobering thought aside, it was a pleasant evening together, and I sat amazed as they rattled off one mnemonic after the other. They even had mnemonics within mnemonics. While one would set the scene for a new problem, the other would rattle off a mnemonic, and explain how they would use it to observe something different to help them get a fuller picture of the problem. They are terrified of having narrow observation within the small window of time they are usually faced with. “But now you notice this in your patient. What do you do now?” From within one letter of one mnemonic, the other would rattle off another mnemonic to use to help jog their observation and thinking to solve that problem, with speed, coherence, and confidence. They would also frequently throw the mnemonics aside altogether, and use other heuristic-based thinking techniques.

They were both in the heat of the discussion, each trying to trip the other up, and the father was attempting to pass on wisdom and experience to his son in a curious Socratic style. I chuckled to myself. I felt if at that instant I were to keel over with some health-related malady, I would be in very capable hands, and someone would learn a new thinking technique in the process.

Modeling Web Applications

When I am testing web applications, I like to analyze the system to find areas of testability, and to identify potential areas of instability. To help remember areas to investigate, I use this mnemonic:

FP DICTUMM

F – Framework (Struts, Rails, Plone, Spring, .NET, etc.)
P – Persistence (Hibernate, Toplink, iBatis, Active Record, DAOs, etc.)

D – Data (database, flat files, etc.)
I – Interfaces (code level, web services, user, etc.)
C – Communication (HTTP(S), web services, SOAP, XMLHTTPRequest, database drivers, etc.)
T – Technology (.NET, J2EE, Ruby, PHP, Perl etc.)
U – Users (Human end users, and other systems or apps that use this system)
M – Messaging (JMS, Enterprise Message Beans, Web Methods, etc.)
M – Markup (HTML, CSS, DHTML, XML, XHTML)

Each of these areas help inform my testing, or provide areas of testability. Understanding the whole picture from the data model up to the HTML displayed in a web browser is important for my testing.