“Why do you like software testing?” is a question that I get asked frequently. A phrase from Richard Feynman comes to mind. When Feynman was asked about how he felt about the reward of his Nobel Prize, he said one of the real rewards of the work he did was “the kick in the discovery.”1 This has stuck with me. As a software tester, I enjoy discovering bugs. I seem to be one of those people who enjoys seeing how a system works when stressed to its limits. I get a kick out of discovering something new in a system, or being one of the first people to use a new system. Scientists like Feynman fascinate me, and a lot of what they say resonates with my thoughts on testing. Software testing can learn a lot from scientific theory; the parallels are very interesting.
Exploratory Testing: Exploring Unintended Test Results
Many of the great scientific discoveries have come about by accident during a typical scientific process of conjecture and refutation. A controlled experiment is a movement towards proving the fallibility of a hypothesis. However, when an experiment has unintended consequences, some scientists do a great job handling things that don’t go according to plan. This often leads to great discoveries.
One parallel to software testing is Ernest Rutherford and his work in nuclear physics. (Someone who spent time “smashing atoms” sounds like someone who might be good at software testing.) Rutherford observed unintended consequences, built on other work and collaborated with peers. He saw unintended consequences in experiments done by his peers, and developed patterns of thought around what was being observed.
The work that eventually led to the discovery of the nucleus of an atom is an interesting topic for software testers to study. If those doing the experiment to push alpha particles through the gold foil had executed the experiment like routine-school testers, and had scripted the test case of firing the particles through the gold foil, and the expected results of all the particles having to go through the foil, would they have noticed the ones that bounced back? If so, what would they have done if the particles that bounced back weren’t in the plan, and weren’t in the experiment test script’s expected results? What if they were so focused on the particles that were supposed to go through the foil that they didn’t notice the ones that did not?
Not knowing exactly how much Rutherford and his colleagues had formalized the experiment, I can’t make any claims on exactly what they did. However, we see the results of the way Rutherford thought about scientific experimentation. What he and his colleagues observed changed the initial hypothesis, and subsequent experiments led to discovering the nucleus of the atom. They had an idea, tested it out, got unintended results and Rutherford explored around those unintended results. He found something new that contradicted conventional knowledge, but transformed the face of modern physics. Like a good exploratory tester, we might infer that Rutherford was more concerned with thinking about what he was doing than in following a formula by rote to prove his hypothesis.
Software testers don’t make discoveries that transform scientific knowledge, but the discoveries that are made can transform project knowledge. At the very least, these discoveries potentially save companies a lot of money. Bug discoveries are hard to measure, but every high-impact bug that is discovered and fixed prior to shipping saves the vendor money. Discoveries of high-impact bugs may be minimized by the team at first, but many times those discoveries are the difference between project success and failure.
James Bach and Cem Kaner say that exploratory testing is a way of thinking about testing. Exploratory testing, like scientific experimentation allows for improvisation and for the exploration of unintended results. Those unintended results are where the real discoveries lie many times in science, and where the bugs often lie in software testing. Detailed test plans and pre-scripted test cases based on limited knowledge may discourage discovery. Tim Van Tongeren and others have done work researching directed observation and the weaknesses associated with it.
One way of thinking about exploratory testing is to see it as a way of observing unintended consequences, exploring the possibilities, forming a hypothesis or theory of the results, and experimenting again to see if the new theory works under certain circumstances. This cycle continues, and testing is as much about making new discoveries as it is confirming intended behaviours . Pre-scripting steps and intended consequences can discourage observing these unintended consequences in the first place. A hypothesis, or testing mission or test case is fine to detail prior to testing, but slavishly sticking to pre-scripted results can stifle discovery.
I have had some testers call exploratory testing “unscientific”. A good scientific experiment to them is about carefully scripted test cases that outline every step and the expected results of that test case. However, many times science doesn’t really work that way. A good deal of care is put into the variables in an experiment, but a lot of exploration also goes on. What is important is not necessarily the formula, but how to deal with unintended consequences. Scientific theory is often more about thinking, dealing with empirical data, and making inferences based on experiments.
Scientific theories go far beyond empirical data, and new experiments confirm and disconfirm theories all the time. Yesterday’s scientific truth becomes today’s scientific joke. “Can you believe that people once thought the world was flat?” As a software tester I’ve known “zero defect” project managers who thought the software was bug-free when it shipped. It wasn’t funny when they were proved wrong, but the software testers were treated like “round ballers” when they provided disconfirming information prior to release.
Good scientists deal with a lot of uncertainty. Good software testers need to be comfortable with uncertainty as well. Software systems are becoming so complicated, it is impossible to predict all the consequences of system interaction. Directed observation requires predictability and has a danger of not noticing the results that aren’t predictable.
Exploratory testing is a way of thinking about testing that can be modelled after the scientific method. It doesn’t need to be some ad-hoc, fly by the seat of your pants kind of testing that lacks discipline. Borrow a little thinking from the scientific community, and you can have very disciplined, adaptable, discovery-based testing that can reliably cope with unintended consequences.