Category Archives: testing

I’m Finding Bugs, But It’s Pissing People Off. What Am I Doing Wrong?

A rookie tester asked me this question. They are experienced in software development, and their grad school work was in a highly specialized area of mathematics. They have been hired to do a very particular type of testing that they are uniquely qualified for. After getting over the mindset shift required to be an effective tester, they were pleased that they were finding bugs that they’d been hired to find. Then a wrinkle appeared. The very people who hired them to find the bugs got angry when they found the bugs they were being paid to find. Sound strange? It might, but it’s actually a common response to effective testing.
Here are some of my clarifying questions:

  • Are the bugs you are finding important, or trivial? Stakeholders on teams can get irritated if you inundate them with only trivial bugs.
  • How are your bug reports? Are they thorough? Do you provide enough information for devs to accurately reproduce the bugs?Programmers get irritated with bug reports they can’t reproduce and track down.
  • How’s your attitude? Do you laugh with glee at the misfortune of the dev team? Are you empathetic, or a condescending jerk? Is your language accusatory. blaming or condescending in any way?No one wants to be around someone who enjoys the schadenfreude.

No, the devs like my bug reports, and I’ve been as low-key and empathetic as possible. These are serious bugs that have probably been in the application since it was released several years ago. I’m tempted to stop logging the kinds of bugs that cause them to get mad at me.

Ok, you’re probably not doing anything wrong. In fact, you are probably doing something right. Don’t stop! You aren’t failing in your work, you are succeeding.

Since they seem to be doing some of the right things and aren’t knowingly or obviously antagonizing stakeholders, I assume that they really are finding important bugs, and that reaction is one telltale sign that they are being effective as a tester. I also encouraged them not to give up, but to do as Peter Block would recommend: move towards the resistance. That is a heuristic that tells us we are doing our jobs, and highlighting the really important problems. The resistance to being confronted with difficult problems that aren’t trivial to solve is just human nature.

As testers, if we are finding good problems, and we’re working with the team to help them, moving away from the resistance is a sure-fire way to relieve the pressure in the short-term, but lose credibility in the long term. If stakeholders realize we can’t be trusted, we’ve lost out ability to effectively test, observe information, and provide information that is useful for them.

I have a personal rule: if I am being pressured not to talk about something, or I feel like I should avoid a contentious issue, that means I absolutely must deal with it and talk about it. I try to be empathetic, understanding, use the right kind of language and not be a jerk, but I bring it up. Every time I do, something important is brought to the attention of the people who need to hear it. They may not like it, and they may not like to see me coming, but they know that I will always tell them the truth.
Early in my career, I was told not to bring up issues that were met with resistance. I was told that was a “career-limiting move.” My career is a testament to the opposite. Whenever I have faced resistance, stuck to my integrity and ethics, and talked about the hard problems the team had been taught to ignore, it has been a career-catapulting move.

So testers, if your work is pissing people off, the problems you are observing and reporting are important, and you aren’t being a jerk, don’t give up. It may hurt to point them out in the short-term, but it pays off handsomely in the long-term.

*Note: This doesn’t just apply to testing work, but any type of work that requires pointing out and helping solve problems. Whatever it is you do, don’t be discouraged if you are working on real problems and people are behaving strangely. Use that resistance to tell you that you are doing something right. The worst thing to do is give up and become silent.

We Need More Metaphors. How About Music?

This month, David Hussman and I collaborated on a piece for SQE’s Iterations newsletter. Check it out here: Iterations January 2009. We talk about the dominance of the manufacturing metaphor in software development, and propose music as an alternative. Be sure to read my “We Need More Metaphors” and David’s “Music and Metaphor” in the newsletter.

Both David and I have been on this line for a while; David has tirelessly presented about his coaching style using lessons from his music producer and professional musician days. I’ve written about music and testing in: Exploratory Testing: Finding the Music of Software Investigation and with Michael Bolton: Testing and Music: Parallels in Practice, Skills and Learning. Watch for more in this space.

Testing is Overrated

I stumbled across a presentation by Luke Francl called: Testing is Overrated. He could have called it: “programmer testing is overrated” but that doesn’t have as much zing to it.

His slides display a refreshing reminder to look for balance in your testing techniques and coverage models. What is especially refreshing is that this is coming from a programmer who is studying testing effectiveness seriously.

Look through the slides and his handout. They provide a nice minimal, complementary and diverse set of testing ideas that when used together can be incredibly powerful.

(If you don’t feel like clicking the links, he recommends a combination of the following: programmer unit testing, skilled manual testing, usability testing, and code reviews.)

Update: Apparently I am behind the times, and Matt Heusser blogged about this very topic in August. It’s worth being reminded of though. 🙂

We Need Testing Example Videos

A few days ago, Michael Bolton, Pradeep Soundararajan and I were sitting around talking about software testing. (Shocking, I know.) Pradeep brought something up that has been on my mind lately: we need more testing videos.

Importance of Visual Examples

Several years ago, I spent a good deal of time with James Bach over a period of a few days. This experience involved a lot of testing exercises, but I learned the most by watching James solve problems and test software.

When I first saw him invoke SFDPOT in real time, while testing an application (instead of as a preliminary thinking tool before hand, which is what I had been doing) I was blown away. I immediately wanted to do that myself. I’d been reading and following James’ work for several years up to then, but seeing him put his ideas into practice was a huge lesson for me. My testing benefited greatly.

Fast forward to my own consulting career. I frequently do live exploratory testing demonstrations during or at the end of my talks. Sometimes they crash and burn and I feel foolish because I said something dumb in the moment and was wrong (Heavens! Can you imagine?) and it was captured on video, and what will people think… but, even when that happens, a small number of people come up and thank me. I’ve even had programmer colleagues attend a talk and say that it all fell into place when they saw me do the demo at the end, and it all made so much sense when they saw it in action.

Video Format

I’ve seen some recordings that are pretty good that use a screen recorder like Test Explorer or BlueBerry Test Assistant. These aren’t bad, but they lack that human aspect and prevent me from completely identifying with what is going on onscreen. I propose that we have demos that use screen recording, but also show different perspectives of the tester sitting at their machine, with audio of the demonstrator expressing their thoughts, perhaps prompted by a narrator. I want to see someone sitting at a computer, and hear them talk about what they are doing and why, as well as the recordings on the screen. I’ve seen some videos similar to this from one of the CASTs, but I want more.

We Need More Testing Videos

I don’t have the time to invest in this right now, so I’m asking people in the testing community to record demos, and make them available. (Maybe we could see if Antony or Rosie could host testing examples for learning, or link to them? Or perhaps someone would like to start a hosting service for example videos?)

I don’t care what flavor of testing you subscribe to, just show us what you enjoy. Show us how you think and test. Demonstrate your skill and use it as a teaching tool. Surprise us. Inspire us to try out your ideas for ourselves.

Exploratory Testing Article

Exploratory Testing: Finding the Music of Software Investigation is an article that explains exploratory testing using music as an analogy.

I first wrote this article over a year ago as an introductory article on exploratory testing. I wrote it with a technical audience who may never have heard of exploratory testing in mind. When one of my programmer friends exclaimed: “I get what exploratory testing is now!” I knew it was time to stop tweaking it. It has languished in my article drafts folder for months, so I decided to publish it myself.

I hope you enjoy it.

UPDATE: In the spirit of the article, and the work that I do, there are reports of the PDF not loading when using an older version of FireFox, with an older version of Adobe Reader. I tested it with Safari, Opera, FireFox and IE 7 with Adobe Reader 8. I get application errors on file load when using FireFox 1.5 and Adobe 6. If you have problems, use “File/Open” within the PDF reader FireFox plugin, which seems to work. That’s the workaround, I’m working on a fix. 🙂

Learning Testing Heuristics

When I was in junior high school, one of my tasks was to learn metric conversions. My science teacher at the time took this task seriously. To help us remember the kilo, hecto, deca, deci, centi, milli prefixes to meter, liter and gram, he had us memorize a mnemonic. The mnemonic looked like this: KHDMDCM, and there was a sort of song that went through it: “King Henry Danced Merrily…something something something”. Trouble was, I barely had the mnemonic memorized in time for a pop quiz, and I couldn’t remember what each letter stood for. I ended up only being able to write the King Henry song out on my test paper and handing it in. I couldn’t do the required conversions such as decimeters to meters, or kilometers to centimeters. My teacher gave me a cold look when he handed the exams back, and admonished me to stop fooling around and actually start trying. Trouble was, I was trying. I just couldn’t get that stupid mnemonic to make sense in my brain.

At home that night, I sang the King Henry song over and over, repeating what each letter stood for. It was hopeless. I had to constantly look at my cue cards.

My older sister happened along and asked: “What on EARTH are you doing?”

I explained.

“Don’t do it that way,” she said, and quickly sketched out a diagram.

She wrote out all the measurement prefixes from largest to smallest, with “meter, liter, gram” in the center, and drew an arrow going up to the left on one side, and down to the right on the other. I stared at the diagram for a few seconds, and she had me reproduce it cold. Within minutes, I had the image memorized. She then taught me how to use it for conversion. It was a snap. On my metric conversion exam later that week, I ploughed through the questions with ease. “Forget King Henry,” was my thought, “this works way better.” And, forget him I did.

However, over twenty years later, I can still scratch out that conversion diagram and convert deciliters to decaliters, or kilograms to grams on the back of a napkin with relative ease.

“Different people learn differently” my father once told me. His first career was a school teacher, and over twenty five years he taught hundreds of different students. He was surprised by little, and was quite pragmatic. If colors helped you learn and remember, use colors. If mnemonics and little songs worked, try them. If images worked, go with the images. Try to involve most of your senses if you want to remember something, he said. Furthermore, if the system you are trying to use doesn’t work for you, make sure you clearly understand the concept, and then create your own. (These didactic sessions usually came from the depths of a newspaper, my Dad gazing at me over the rim of his glasses with CBC Radio blasting in the background.)

When I started working with James Bach, he talked about using mnemonics for testing. I was familiar with, and had used most of his published mnemonics, but I had never memorized them. I used them more for analysis prior to testing. They seemed suited for analysis and test planning, and I was able to use them to generate extremely effective test strategies and plans.

Then one day I saw James at work, testing software. He was thinking out loud, using test heuristics, rattling off testing mnemonics, and simultaneously designing and executing tests under each letter of a mnemonic in real time. He would pause periodically to point out his findings or jot down notes. He would generate and execute many test ideas in different categories in a short period of time. This resulted in surprisingly thorough testing, with test ideas no on else had thought to try. James’s testing provided a veritable treasure trove of bugs within minutes. Whoa.

James was using SFDPOT (San Francisco Depot), and he would say the first letter of the mnemonic out loud like this:”S – SYSTEM!” Then he would mutter and brainstorm and figure out ways to test the system part of the application. He might alter the files in an installed app, or remove dependencies. He would use system tools to monitor memory use and CPU usage for example, while testing. If he saw that memory use jumped up when doing a particular action, he would repeat it rapidly to see if memory use kept increasing, and if it fell back down to where it was before. He would uncover installation issues, improper dependencies, memory leaks, and all sorts of things in a very short period of time.

Once he was satisfied with those tests he would sound out “F- FUNCTION!” Then he would generate a few ideas of functional tests, and he would examine areas that seemed out of the ordinary and push them. If he found an awkward workflow, he would note it down. If he also noticed memory leaks or CPU usage spikes, he would note what he was doing that caused them. He would continue on through Data, Platforms, Operations and Time, then he would stop testing, consult his notes and start logging bugs.

I’d never seen anything like it, and I knew right then I had to master his approach. But memorizing mnemonics was a bit of a blind spot for me. I needed to figure out how to memorize those mnemonics, so I too could quickly slice through a testing problem (any testing problem) with thorough ease. However, the spectre of King Henry loomed above me, reminding me of my early mnemonic memorization failure.

James peered down at me over the rim of his glasses: “You need to memorize testing mnemonics.”

His tone was casual, but firm. To him it was like telling me I needed to learn to cross the street. Obvious.

I complained about how I often found it hard to memorize mnemonics. Then, much like my sister did back when I was in junior high, James told me not to do it that way anymore.

“That’s because you are memorizing someone else’s. You need to create your own. If they are yours, you’ll remember them.”

He then spent time with me helping me practice memorizing a mnemonic, using the mnemonic to test in real time, and then encouraged me to personalize it and make it my own.

It turned out I could learn new tricks. After all, I had used mnemonics as a student that were quite useful and effective. Who can forget SOHCAHTOA, an effective way to remember how to calculate the sine, cosine and tangent of an angle? I remembered memorizing that mnemonic and using it to great effect. “Soak-a-toe-ah” I repeated in a unique cadence, and imagined a big toe in a wash basin. The imagery and the rhythmic cadence of “soak-a-toe-ah” helped me remember the mnemonic, and practice helped me remember the application. I could memorize testing mnemonics just as easily, and banish King Henry from my memorization nightmares. It turns out that motivation and seeking results can go a long way to memorizing a mnemonic.

I was faithful to James’s advice, and mnemonics have served me well. While I use other people’s mnemonics in my own work, I find I return to my own more often. I’ve added mnemonic memorization and creation to my testing thinking repertoire. I use ones from others that are particularly effective, and I create my own as I need to as well.

When I teach testing techniques, I find, as my father did many years ago, that some thinking models appeal to different people in different ways than others. Mnemonics and other thinking aids don’t need to be lists, they can be imagery, visual patterns, sounds, or anything else that helps you remember to be more thorough. Any thinking model that gets the job done is valid. Furthermore, using different thinking models along with memorized test heuristics helps me analyze them, enhance them, and learn more about my own work. It also means my implementation looks different from someone else’s, based on my own experience, insight and skill. Now, I don’t just use test mnemonics to help plan, I use them during test analysis and planning, and during test execution.

EDIT Update: I SLICED UP FUN is a popular mnemonic I created. I developed it first for myself, then shared it with my team, then with everyone else. It has become quite popular on mobile software development projects the world over.

Are you curious about using mnemonics in your work? If so, imagine me peering over my glasses at you, expecting you to make them your own.

Well, well, well

Living in Alberta, I do testing work in the energy industry, or with companies that support the energy industry with software or services. Alberta is Canada’s energy capital, the largest oil and gas producer in the country. Rudyard Kipling said when he visited Alberta that we “…seem to have all hell for a basement”.

As a kid growing up in the ’70s and early ’80s, it seemed every farmer’s field in our area had a pumpjack servicing an oil well, or had a gas well feeding into a pipeline. In the energy industry, the term “well” is common, but vital. Wells are one source from which oil and gas flow, so a lot of resources are put behind drilling to find new sources, or to extract more oil and gas from known producing reservoirs beneath the surface.

I was recently testing an application that serves this industry. It had a neat search function, so I typed in the name of one of my test wells I had set up. I immediately got an error, with messaging informing me that the application could not complete my search because it had “common” words in it. The example it gave was “the”, and the messaging said to try it again without a common word. I immediately suspected that it was filtering out the word “well”, so I repeated the search without the term “well”, and it worked like a charm.

I logged the issue and the developers investigated the cause. (I was off-site, so I didn’t get all of the details I normally would.) They told me that this problem was due to the database server configuration that filters out common words on queries. The software that managed the database had a feature that they tought would be helpful in many contexts. Sadly, this feature the database vendor thought would be helpful caused problems in our context. An assumption with the term “well” as always being a common word for all contexts caused this feature to be useless in our context. Context is important, and assumptions are frequently a source of bugs.

In what other contexts would this be a problem? I can think of at least one. If I had this feature turned on, my database and I worked for a company that managed water sources, I imagine I would run into problems as well with searching in this way, since water wells are also common, particularly in rural areas. But this is a context where the problem is going to be obvious.

When I saw the error message, I used inference to generate the best explanation. My inference immediately told me that it was probably filtering out the term “well”, so I tested a conjecture (“they are thinking well is a common word”) and as far as I can tell, it turned out to be the case. But what if I was in an industry that was one step away, where people aren’t talking about wells, and reciting well ids, or talking about drilling, extraction, and other tasks related to wells? That’s where these kinds of problems turn into what might seem to be a sporadic or “unrepeatable bug”. We know there is a problem, but it isn’t obvious to us because our context doesn’t make it obvious. It would be obvious to someone with experience in the right context, it just isn’t obvious to us. Our observations are filtered due to our context, our knowledge and our biases. How do we get beyond this when dealing with a problem that seems mysterious to us?

Here is an example that I am dreaming up to fit this scenario. Let’s pretend that we have written a cool search tool that is fast, accurate and works well in corporate environments. We have sold this tool to companies in the transportation and financial industries. We also sold it to a couple of companies in the energy industry. Our tool works like this: it is installed on application servers in the corporate head office and is managed by the IT department. It uses a database to store information gathered from disparate sources, and updates the data found on the network, intranet, etc, on a set time. It might update every half hour, or on a nightly job. Our customers are thrilled with the tool, they now have a simple interface into this scattered data, but a couple of them are complaining about not being able to find a small amount of critical data. It appears to be lost. You, the reader are the tester working for the search company and are helping investigate this problem.

The team is frustrated, they just can’t get a repeatable case with the information they have. You have tried all your existing test data that you’ve generated and normally use with regression tests, but the software works fine in all your tests. The team feels that they have exhausted testing ideas, but a couple of clients are still complaining, and getting less patient with your company.

Given the information you know from this post, what would you do? Feel free to email me your ideas, and I’ll post a follow up with some of your suggestions as well as my own.

Testing an Application in Layers

There is often debate about test automation versus manual testing. When I think about testing, I look at an application in 3 broad layers: the code (on the machine side), the system (where the finished software lives), and the visible layer, or how the software is used from an end user’s perspective. I often call this visible layer the social context because of the environment much end-user software is used in. When we spend a lot of time in one context, testing starts to specialize because we concentrate on part of the picture.

When we view an application from the source code view, the testing is dominated by automation. When we look at the system context (as some of my operations friends do), testing involves integration in a system, and testing hardware, firmware, drivers, etc. to make sure the software gets served up correctly. Automated testing tends to get more complex the more we move from the code to the user interface. Attempting to emulate user actions is difficult, and high-volume automated functional tests can involve massive amounts of automated test code. This can be problematic to maintain. Sometimes functional tests become so complex they involve as much test code as the software they are testing has to serve up a component, plus test data generation code, as well as the code that attempts to emulate user actions.

Personally, I agree with Cem Kaner and call automated testing “computer-assisted testing”. The computer is a tool I use in conjunction with good manual testing. Until machines are intelligent, we can’t really automate testing. We can automate some aspects of testing to help maximize problem-solving efficiency.

Traditionally, software testers tend to have a handle on the social context, or how the software is used in a business context. As a result, much conventional testing is focused on the visible layer of the application. I tend to prefer testing at various layers in an application. I value testing components in isolation as well as testing software within a system. There are advantages and drawbacks to both. While I value isolation, testing, particularly brain-engaged manual testing at a visible UI layer has merit. Sometimes testers focus a great deal on testing in this context when component isolation might be more efficient. Often, traditional testers hope for an automated testing tool that can do the work of a tester. I’ve yet to see this occur successfully, but there are still tasks that can be automated to help testing efforts that are a big help.

Frequently there are bugs that are difficult to track down that crop up in the visible layer of the application. These are due to the visible application at runtime becoming greater than the sum of its parts. There is a kind of chaos theory situation that occurs due to the application being used in a way that the underlying code may not be designed to handle. By the time a minor fault at the code level bubbles up to the UI, it may have rippled through the application causing a catasrophic failure. Unfortunately, these kinds of usage-driven faults are problems automated tests at various layers do not tend to catch. Often it is some sort of strange timing issue as I note in this article. Other times, it’s due to actions undertaken in a social environment by an unpredictable, cognitive human. These variable actions motivated by inductive reasoning, driven by tacit knowledge are difficult to repeat by others.

Focusing too much on one testable interface in an application can skew our view. If we view the application by the code most of the time, we have a much different picture than if we view it through the UI. Try flipping the model of an application on it’s side instead of viewing it bottom up(from the code), or top down(from the ui). You may discover new areas to test that require different techniques, some of which are great candidates for automation, while others require manual, human testing.

Tim’s Comments on Software Testing and Scientific Research

Tim Van Tongeren commented on one of my recent posts, building on my thoughts on software testing and the philosophy of science. I like the correlation he made between more scripted testing and exploratory testing to quantitative vs. qualitative scientific research. When testing, what do we value more on a project? Tim says that this depends on project priorities.

He recently expanded more on this topic, and talks about similarities in qualitative research and exploratory testing.

Tim researches and writes about a discipline that can teach us a lot about software testing: the scientific process.