Test-Driven Development and Exploratory Testing

I’ve spent time working on teams practicing Test-Driven Development. Since I am a tester, and the practice has the word “test” in it, I’d only been on Agile teams for a short time before I was enlisted by programmers to help them out with test idea generation during Test-Driven Development.

At first, it felt a bit foreign, particularly when the programmers were using tools and programming languages I wasn’t that familiar with. In fact, it didn’t really feel like testing at all; it seemed like a programming style that was heavily influenced by object-oriented programming techniques. Once in a while though, it really felt like testing, the kind of testing I was familiar with. The programmers would turn to me to get me to drive when pairing, and were acutely interested in test idea generation. This was where using exploratory testing techniques, particularly rapid testing techniques really shone. Since we were developing new features quickly, the faster I could generate useful and diverse testing ideas, the quicker we could generate tests, strengthen the code, and improve the design. When we went back to developing a new feature though, I again felt out of place, particularly when I distracted and frustrated development at first.

I decided that the only way to understand Test-Driven Development was to immerse myself in the practice, so I paired with an expert programmer who had experience teaching TDD. I describe some of my experience in this article: Test-Driven Development from a Conventional Software Testing Perspective, Part 1. I found that I was still struggling with parts of TDD, and I felt like I would contribute very little at some points, and contribute a great deal at others. William Wake cleared this up for me. He explained two phases of Test-Driven Development: the generative phase and elaborative phase.

Generative tests are the first tests that are written, primarily about design. They’re “shallow sweep” tests that yield the initial design. Elaborative tests are deeper, more detailed tests that play out the design and variations.

Transitions between these phases can occur at any time, which is why I hadn’t noticed the change between my testing idea generation providing more or less value. At some points I would actually distract the programmer with too many test ideas, and at others, they would sponge up everything I could generate and would want more. I had to learn to watch for patterns in the code, for patterns in our development, and to work on my timing for test idea generation.

When we were clearly in the elaborative phase, our testing was much more investigative. When we were in the generative phase, it was much more confirmation-based. In the elaborative phase, we were less concerned with following TDD practice and generating code than we were with generating test ideas and seeing if the tools could support those tests or not.

To understand more, I decided to do TDD myself on a test toolsmith project. I describe those experiences in more detail here: Test-Driven Development from a Conventional Software Testing Perspective, Part 2. This experience gave me more insight into the testing side of TDD.

One day while developing in the generative phase, I was suddenly reminded of pre-scripting test cases, a practice I did religiously in the beginning of my career before fully embracing investigative exploratory testing. Instead of recording tests in a document or in a test case management system, I was pre-scripting them in code. Instead of waiting for a build from the development team to run the scripts, I was running them almost immediately after conceiving them and recording them. I was also recording them one at a time and growing a regression test suite at the same time as creating scripts. I realized that an assertion in my tests was analogous to the “Expected Results” section of a test script, an idea popularized by Glenford Myers in his book “The Art of Software Testing”, published in 1979.

When I moved to the elaborative phase and used different kinds of testing, glaring weaknesses in my test-driven code became apparent. In the elaborative phase, I didn’t care about expected results nearly as much as the unexpected results. The testing here was investigative, and if I pulled off my programmer hat long enough, I could go quite deeply with exploratory testing techniques. When a weakness was exposed, that meant I had missed test(s) in the generative phase, and I would add new test(s) to cover those missed cases to my unit testing code.

The main difference between what I was doing in code and what I would normally do with a program was the interface I was using for testing (I was using code-level interfaces, not the GUI), and I was the one fixing the code whenever I found a problem. Since the tests were highly coupled, and expressed in the same language as the program code, test case execution and maintenance wasn’t as disconnected.

I was also less concerned with my test case design up-front in the elaborative phase. It was much more important to use the programming tools to allow me create different kinds of test cases quickly, or on the fly. Once I found something interesting, and decided to record a test as a regression test, I would convert the test into a proper unit test with setup method, a test method with an assertion, and a teardown method.

Now that I’ve had some experience myself with TDD, and have worked with skilled practitioners, I have a better idea of what it is, and how it can relate to exploratory testing and scripted testing. There are a lot of elements of pre-scripting test cases in the generative phase, or scripted testing, including some of the drawbacks of that practice. Over several releases, pre-scripted test cases tend to have an increasing maintenance cost. (Note, Story-Test Driven Development is another form of pre-scripted, confirmatory testing which can exacerbate the maintenance problems we had with pre-scripted tests in pre-Agile days. They can be extremely maintenance intensive, particularly over long projects, or over multiple releases, robbing resources from new feature development and investigative testing activities.)

The elaborative phase was a natural fit for exploratory testing, and a great place for non-technical testers to jump in and help generate testing ideas. The challenge with this is figuring out what is good enough, or when to stop. Investigative testing is only limited by the imagination of the people involved in the activity. When there are two of you, a tester and a developer, it is easy to quickly generate a lot of great ideas.

It is also common for TDD practitioners to skip the elaborative phase for the most part. In this case, TDD is almost exclusively about design, with little testing going on. The “tests” are mostly a type of requirement, or examples. I’ve also seen some programmers abandon elaborative tests because they failed, while the generative tests passed. Others just didn’t do much elaboration at all, partly because they were so focused on velocity that they wanted to get a story written as quickly as possible.
In other cases, the phases were more blurred, particularly with programmers who felt that testability improved their design, and were open to a lot of different kinds of testing techniques. In fact, because the feedback loop with pre-scripted generative tests is so tight, a programmer who is experiencing flow may be utilizing exploratory thinking while developing new features.

I talk more about some of my opinions on TDD from a tester’s perspective here: Test-Driven Development from a Conventional Software Testing Perspective, Part 3. This reflective article spawned some interesting ideas, and while recording the skepticism and cautionary points, I had a lot of ideas for new areas of work and research.

Several TDD practitioners have approached me about exploring the elaborative side of TDD with skilled exploratory testing to address some of the problems they were having with their code. One area I consistently hear about are tests that yield a brittle design. Once the code is put into a production-like environment, testing revealed a lot of errors they felt would have been better addressed during development. The programmers felt that with more exploratory testing with more elaboration would combat this. Using more elaborative tests, more often, might be one way to maintain exploratory thinking in an emerging new design. Instead of a narrow focus of “write a test, run it, when it fails, write code so it passes”, they felt that exploratory testing would expand their exploratory thinking while programming, with a better overall design as the desired result. Rapidly transitioning from generative design thinking to elaborative, exploratory testing is one area to explore.

This intersection of investigative testing and programming holds a lot of promise for practice and research. I encourage interested testers and programmers to work on this together. I’ve certainly learned a tremendous amount by working in this space, and the testing community has benefited from the subsequent improvement in practices and tools. In fact, recently as I was rapidly writing web services tests in Java/JUnit, I recalled how much work it was eight years ago for a testing department to create a custom test harness in Java before we could even write tests.