Category Archives: test scripts

Getting Started With Exploratory Testing – Part 3

Wean Yourself off Scripted Tests

Scripted tests, or test cases that are written in advance, contain procedural steps to execute, and usually have some sort of “expected results” at the end are very common. If you are a tester who is used to only using these kinds of test cases, transitioning to exploratory testing has special challenges.
One way to get started with exploratory testing is to change the way you implement scripted tests, with two goals in mind:

  1. to get a feeling for exploratory testing
  2. to be able to engage in testing without needing pre-recorded test scripts

Some testers can find it overwhelming and can almost feel a sense of panic when they consciously try out exploratory testing. Exploratory testing is actually a natural activity for us to do, but sometimes we are so conditioned to follow a script, and only test what we’ve been told to test, it can be a difficult transition to make at first. There are some useful things to remember:

  • If you feel overwhelmed by possibilities, stop, gather your thoughts, then pick one simple top priority item to test.
  • There is no right or wrong way of practicing exploratory testing; your own unique creative and idea generation methods are a source of strength.
  • Minimal planning can go a long way. To start, set out a time period (an hour might be a good place to start), and an area of focus in an application.
  • For your self-allotted time box of exploratory testing, try not to multitask. Stick to your area of focus, and resist the urge to check email, answer colleagues’ questions, or test other applications. If you find areas to explore outside of your area of focus, quickly take note of them, and return to them after your time-boxed testing session.

One way to do this is to encourage yourself to add more variability when exercising a test script. Elisabeth Hendrickson talks a lot about adding variability to testing in her work. One way to consciously add variability is to change the program inputs that are recommended in the test script you are following. You can change them slightly, or dramatically, feel free to experiment with different types, or use some of your “old favorite” inputs, write a script to generate test data, or use something like PerlClip.

Another way to consciously add variability into your test script is to vary the steps. Here is an analogy. A friend of mine is an accomplished musician with strong technical skills, but struggles with improvisation. Here is an exercise we did recently:

  1. I recommended that she take a simple music score (like Mary Had a Little Lamb), and try to play it note for note. I wanted her to play a rendition that she felt was as close to the way the author intended the sheet music to be interpreted.
  2. Next, I asked her to add ornamentation to her playing. (This is subtle variation, or a slightly unique style that comes out when playing an instrument, which tends to be unique from musician to musician.)
  3. Once she had done that, I asked her to play the song again, but to ignore some of the bars in the sheet music. I asked her to anticipate ending up on an important, signature part of the song so it was still recognizable, but encouraged her to do whatever came to mind to get there. It could be silence with counting and waiting; it could be one extended note; it could involve running up and down an appropriate scale; trying to play what they thought should be there from memory, or what was going on in her head.
  4. With that out of the way, I told her to throw caution to the wind and play with complete freedom. I asked her to play for roughly the same amount of time as the song might take, but to play whatever came to mind.

I asked her to record and send me her versions of the song at each step. The results were surprising and satisfying. In step 1, the song sounded as you would expect, but was a bit wooden, or forced, like a computer had played it. In step 2, the performance came alive. In step 3, there were flashes of brilliance and the odd spot where execution went off the rails. Part 4 was a huge surprise. My friend had been listening to progressive heavy metal earlier in the day, and the piece she came up with while improvising didn’t sound like Mary Had a Little Lamb at all. It sounded like part of a Dream Theater song. The notes were there, the scales were there, in fact, most of the framework of what Mary Had a Little Lamb is composed of was there, but it wasn’t recognizable.

However, it was incredibly effective as performed music. It was interesting, entertaining, creative, and it flowed with an effortlessness that resonates and connects with a listener. “Flow” is an important concept both for musical improvisation, and for exploratory testing.

With my musician friend, my goals were to get her to feel “flow” when playing her instrument, and to meld her thought processes with her physical manipulation of the instrument. Most importantly, I wanted her to let go of her inhibition when playing, to not worry about breaking the rules, and to just be free and have fun while achieving a goal. “Flow” is not only important in creative endeavors, it applies to any kind of human/computer interaction. This article is a good overview of the concept from a gaming perspective.

Do you have enough information to apply this kind of thinking to your testing work? If you’re still stuck, here are some things to consider:

Contrary to some perpetuated testing folklore, you do not need “Expected Results” to be written down prior to testing. James Bach says:

The expected result thing is interesting because we have a million expectations, mostly unconscious and emergent. To suggest that there are only one or two expected results is absurd. What I say is not that you need an expected result for a test. What you need is a reasonable oracle. This is a broader idea than expected result. By the way, your oracle is not necessarily needed before the test is executed. It can come at any time, even a week later.

Note: An “oracle” is defined as: “…a principle or mechanism by which we recognize a problem.” Also notice that in my example above, the “expected results” are not written down in the sheet music, they are implied expectations of the performer and the listeners. Not having them written down is no more an impediment to testing than it is not an impediment to performing music.

You do not need formal requirements, or a complete specification to test. As a software user and tester, you have what James Bach calls “reasonable expectations”. Michael Bolton has a nice article that addresses this issue: “Testing Without a Map“. You can use some of your scripted test cases as a place to start, but one goal for your exploratory testing should be doing something like what Michael describes here.

To start weaning yourself off of scripted tests, try to repeat what my musician friend did above, but with test cases you are familiar with. If you get stuck trying to transition away from a scripted test, here are some things to try.

  • Execute a test script, but stop part way though, and interact with the application at whatever point you are at, then come back to the test script. Does following the script feel different? Take note of that feeling, and try to replicate the feeling you get with exploratory testing more often in your testing execution, and when thinking about testing.
  • Stop part way through the script, and engage in another activity. Stare out the window, take a short walk, or read a testing article, and then come back to your test script. Instead of following the rest of the test script steps, focus on the first thing you see in the application, and interact with that component.
  • Memorize the test script, re-run it, and then get creative in your execution of it. Can you change it so much that it still meets the intent or goal of the script, but the original writer may not recognize it?
  • Try to be as observant as possible. Can you see anything different that you might not have noticed before? Stop and take notice of what the application is doing. Does anything feel different or suspicious to you? Is there a blink, or an oddity in the way a screen loads? If something feels funny, this is a good sign to investigate that thing that seems a bit off. Embrace those feelings when they come, that is often a sign that you are almost subconsciously observing something different.
  • Research other ways of interacting and implementing tests, and pre-plan a completely new strategy for testing that application. Can you use a different testing tool? Execute those tests through a different interface than the user interface? Once you have a new plan of attack, follow the intention of the script, but not the steps of the script using a different testing strategy or technique.
  • Focus on observing slight details when testing, especially ones you didn’t see before, and capitalize on exploring them. Practice developing your observation skills away from the computer, and apply your learnings to testing.
  • For something completely different, try testing James Lyndsay’s black box test machines. There’s nothing quite like testing something that has no written documentation at all to go on.

With time, you should find that you no longer need pre-scripted tests at all to begin testing. All you will need is software to test, and your testing ideas and thinking skills. If you can learn to alternate between both scripted and exploratory testing, your testing and your thinking about testing will change. I hope this will help testing become more interesting for you, particularly if you feel locked in scripted testing mode. I also hope that if you practice and think about testing, your confidence in your abilities as a tester will grow.

Placating with Paper

Accountability. This is often a scary word for some people. Sometimes when we are fearful of the results of a decision, or of our work on a project, we will look at what we can do to offload responsibility. “At least I won’t be blamed if anything goes wrong.” might be the thinking. It’s hard to constantly offload onto other people though. After a while, we’ll run out of team members to blame, and we probably won’t be too popular. One easy way around this is to offload accountability onto an inanimate object: paper. Paper often comes in the form of specifications, design documents, test plans, test cases, or as I see more lately, proof of adherence to a popular development process. Paper might be substituted when we need to do something difficult, or provide reasons why a product isn’t working, or when we need to say something others may not want to hear. Paper with fancy words and jargon is often accepted as a short-term solution to project problems. Trouble is, what does it help in the long-term, when you are being paid to deliver software?

In one case, paper provided hope to end an unpleasant release. The development team had just come out of a meeting with our sales team and a couple of important clients. They were quite clear that we needed to deliver a particular feature in our next release. Unfortunately, it wasn’t working properly. Whenever we tested this new feature, the application crashed in a spectacular manner. There had been two attempts at redesigns, and everything else in the release seemed solid. The programmers were tired and demoralized, management didn’t want to hear about any problems anymore, and the testers were tired of testing and feeling the weight of the release resting on their shoulders. They felt singled out because whenever they logged a bug report, the rest of the team would accuse them of holding up the release. “Jonathan, why are we testing this feature in this way?” the test lead asked, and pointed to the specification document.

“We’re testing this because this is the most important feature in the release. Without this feature, our sales team is going to struggle, and our customers are going to be upset. Remember what we talked about in the meeting with both the customer and the sales team?” I asked. “Sure I do, but look at the spec. It doesn’t spell out what the customer asked for. If we test to the spec, we can ship! The tests will all pass if we only test according to the spec.”

The lead tester had found a loophole. The thinking was that if we tested only to the spec, and followed the document to the letter, we could pass the tests that were currently failing, and management would happily release the software. Everyone would be relieved, and furthermore the lead pointed out, who could blame the testers if anything went wrong? The design spec would be at fault, not the testers or developers. “But it flat-out doesn’t work!” I retorted. “Sure it does, according to the design spec.” they replied. “But the customer can’t use a design spec to do their jobs, can they? If our software doesn’t do what they expect it to do, are they supposed to print out the spec, clutch it to their chests and draw comfort and dollars from it? The spec is meaningless to the customer when they are trying to use our software. They need this feature to work in their context, to help them get their work done. They are relying on us to deliver software, not a piece of paper with a hollow excuse.”

I was a bit shocked at what this test lead proposed at first, but dealing with pressure in this way is not uncommon. A paper document can carry a lot of weight in an organization. People can draw a false sense of security from some official looking prose wrapped in corporate logos. I encouraged the tester to not fall for quick fix technicalities, especially if she wants to be a tester for very long. Fight the good fight, and while it may be hard in the short-term, the payoff in the long run is great. Happy customers, and the respect of your peers are just two payoffs. Being able to look at yourself in the mirror every day and know you have your integrity in tact is also something important, and no project deadline is worth sacrificing that for. A tester with a reputation for having no integrity will not be employable in the long-term.

On another project, paper was used in a different way. I was on one small team, and we met with other teams in the company periodically to share ideas. Another team came to a meeting and crowed about the “traceability” of documentation they were creating on a large project. We were on a small project at the time, and we were minimalists with documentation, preferring to deliver working software that could be deployed into production every two weeks. “That’s fine, but you don’t have traceability like we do!” they said. They bragged that for every method of code that would be written, there would be three documents generated as “project artifacts”.

They met with us again in a few months. We had delivered two projects that were helping the company make more money, and they were still talking about documents. “Can you show us the code?” we asked. “We’re still in the requirements-gathering phase”, they snapped. They were too busy to code, instead, they were writing documents — documents about the design, documents about a myriad of tools they could choose, and more documents documenting other documents. The company had spent several million dollars that year, and there wasn’t one line of code to show for it. They had so-called “traceability” though, with project documents, and committees reviewing documents, and committees reviewing other committees. An entire infrastructure to support this process was developed. There were literally teams of people working full-time churning out documents, but finding a programmer in all of this was not an easy task. Expensive tools to aid in paper generation were purchased, more people were hired to keep up, and project budgets were sacrificed on the altar of the great “project artifact” gods. (All stages of each properly documented and “signed off” on of course.)

Through all of this, management was placated with a blizzard of paper. “Do you have a demo yet?” they would ask, only to be inundated with mountains of UML diagrams, meeting minutes, decision matrices and vendor white papers. This satisfied management in the short-term. Look at all this paper – lots of work must be getting done.

In the end, paper didn’t deliver. There came a time when documents just didn’t cut it anymore. Management wanted a return on investment. They didn’t care about the development process du jour, detailed design documents, and requirements changes and budgets signed off in triplicate. They wanted to see the project working in production, helping them realize their business goals of a return to investors. There wasn’t much of a product there though – there were lots of documents, but precious little in the way of tested, delivered software.

The project went several years, and millions of dollars over budget. It was also a public embarrassment for the company. People identified with that project were blamed, and association with that project was an impediment to finding work with other companies who had heard about it through the grapevine.

What went wrong? The vendors, consultants and employees knew they could placate management with paper, and management didn’t demand a working solution early enough. This project could have done well, had they released working software incrementally, rather than churn out ever-growing mountains of paper. Instead of taking personal responsibility, the people working on the project placated customer demands with paper.

What were they thinking? Maybe they didn’t really know what they were doing. Maybe they hoped the programming elves would rescue them if they created enough documents. Maybe they got swept away in groupthink, and followed a process instead of working to deliver software. Maybe they collectively thought that a colleague would take responsibility. Maybe they just didn’t care. I don’t know.

The customer unwittingly paid people to generate a lot of paper, instead of getting them to write and demonstrate working software. They accepted paper designs instead of code that could be demonstrated. They accepted paper over skill, and they accepted paper instead of accountability and working software.

Accountability. Demand it from yourself, and from others. Placating with paper may get you through the short-term, but sooner or later you have to demonstrate your skill. Skill can be demonstrated individually as a programmer, as a tester, a technical writer, a project manager, and demonstrated collectively as a team. It will ultimately be demonstrated in your product. Does what you produce match your claims? If it doesn’t, a document and a nice explanation may get you out of a jam in the short-term, but in the long-term, you have to deliver what the customer needs. The customer won’t blame the document – they will blame you and your company, and move on to a competitor’s product. Where does that leave you?

Procedural Test Scripts

Cem Kaner has sometimes called detailed manual procedural test scripts “an industry worst practice”. I tend to agree. At one time I thought they were a good idea, but they lead to all kinds of problems. One is a lack of diversity in testing, another is that they become a maintenance nightmare and rob time that could be spent actually testing the software with new ideas. Another problem is that we usually write them off of requirements which narrows our focus too much, and we write them early in a project when we know little about the product. But I’m not going to get into that in this post. Instead, I’m going to describe a recent conversation that outlines why we as testers should question why we do this practice.

I was recently discussing the creation of procedural test scripts prior to testing with developers. They were skeptical of my views that pre-scripting detailed manual test cases are a scourge on the testing world. “How will testers know what to test then?” I replied: “Good testers will use their judgment and skill. When they don’t have enough information, they will seek it out. They will utilize different testing heuristics to help meet the particular mission of testing that is required at the time. They will use information that is available to them, or they will test in the absence of information, but they will rely on skill to get the job done.” This didn’t resonate, so I came up with an equivalent for developers. Here is the “procedural development script” that follows what is so often done in testing. (To make it more authentic, it should be written as long before the actual development work is done as possible).

Development Procedural Script:

Purpose: write widget foo that does this business functionality


  1. Open your Eclipse IDE. Start/Programs/Eclipse.
  2. Select your workspace for this project.
  3. In the package explorer, create a new java source code file.
  4. Begin typing in the IDE
  5. Use the such-and-such pattern to implement this functionality, type the following:
         public void <method name>(){

The development manager chuckled and said he’d fire a developer who needed this much direction. He needs people with skill that he can trust to be able to program in Java, to use their judgment and implement what needs to be done under his guidance. He would never expect developers to need things spelled out like that.

I countered that the same is true of testing. Why do we expect our testers to work this way? If we scoff at developers not needing that kind of direction, why do we use it in testing? Why do we promote bad practices that promote incompetence? Testers need to have skill to be able to figure out what to test without having everything handed to them. If we can’t trust our testers to do skilled work without having to spell everything out first, we need to get better testers.

Developers and business folk: demand skill from your testers.

Testers: demand skill from yourselves.

Are you a tester who wants to improve their skills? Cem Kaner’s free Black Box Software Testing is worth checking out.

Testing Debt

When I’m working on an agile project, (or any process using an iterative lifecycle), an interesting phenomenon occurs. I’ve been struggling to come up with a name for it, and conversations with Colin Kershaw have helped me settle on “testing debt”. (Note: Johanna Rothman has touched on this before, she considers it to be part of technical debt.) Here’s how it works:

  • in iteration one, we test all the stories as they are developed, and are in synch with development
  • in iteration two, we remain in synch testing stories, but when we integrate what has been developed in iteration one with the new code, we now have more to test than just the stories developed in that iteration
  • in iteration three, we have the stories to test in that iteration, plus the integration of the features developed in iterations that came before

As you can see, integration testing piles up. Eventually, we have so much integration testing to do as well as story testing, we have to sacrifice one or the other because we are running out of time. To end the iteration (often two to four weeks in length) some sort of testing needs to be cut in this iteration to be looked at later. I prefer keeping in synch with development, so I consciously incur “integration testing debt”, and we schedule time at the end of development to test a completed system.

Colin and I talked about this, and we explored other kinds of testing we could be doing. Once we had a sufficiently large list of testing (unit testing, “ility” testing, etc.), it became clear that the “testing debt” was more appropriate than “integration testing debt”.

Why do we want to test that much? As I’ve noted before, we can do testing in three broad contexts: the code context (addressed through TDD), the system context and the social context. The social context is usually the domain of conventional software testers, and tends to rely on testing through a user interface. At this level, the application becomes much more complex, greater than the sum of its parts. As a result, we have a lot of opportunity for testing techniques to satisfy coverage. We can get pretty good coverage at the code level, but we end up with more test possibilities as we move towards the user interface.

I’m not talking about what is frequently called “iteration slop” or “trailer-hitched QA” here. Those occur when development is done, and testing starts at the end of an iteration. The separate QA department or testing group then takes the product and deems it worthy of passing the iteration after they have done their testing in isolation. This is really still doing development and testing in silos, but within an iterative lifecycle.

I’m talking about doing the following within an iteration, alongside development:

  • work as a sounding board with development on emerging designs
  • help generate test ideas prior to story development (generative TDD)
  • help generate test ideas during story development (elaborative TDD)
  • provide initial feedback on a story under development
  • test a story that has completed development
  • integration test the product developed to date

Of note, when we are testing alongside development, we can actually engage in more testing activities than when working in phases (or in a “testing” phase near the end). We are able to complete more testing, but that can require that we use more testers to still meet our timelines. As we incur more testing debt throughout a project, we have some options for dealing with it. One is to leave off story testing in favour of integration testing. I don’t really like this option; I prefer keeping the feedback loop as tight as we can on what is being developed now. Another is to schedule a testing phase at the end of the development cycle to do all the integration, “ility”, system testing etc. Again I find this can cause a huge lag in the feedback loop.

I prefer a trade-off. We have as tight a feedback loop on testing stories that are being developed so we stay in synch with the developers. We do as much integration, system, “ility” testing as we can in each iteration, but when we are running out of time, we incur some testing debt in these areas. As the product is developed more (and there is now much more potential for testing), we bring in more testers to help address the testing debt, and bring on the maximum number we can near the end. We schedule a testing iteration at the end to catch up on the testing debt that we determine will help us mitigate project risk.

There are several kinds of testing debt we can incur:

  • integration testing
  • system testing
  • security testing
  • usability testing
  • performance testing
  • some unit testing

And the list goes on.

This idea is very much a work-in-progress. Colin and I have both noticed that on the development side, we are also incurring testing debt. Testing is an area with enormous potential, as Cem Kaner has pointed out in “The Impossibility of Complete Testing” (Presentation) (Article).

Much like technical debt we can incur it unknowingly. Unlike refactoring, I don’t know of a way to repay this other than to strategically add more testers, and to schedule time to pay it back when we are dealing with contexts other than program code. Even in the code context, we still may incur testing debt that refactoring doesn’t completely pay down.

How have you dealt with testing debt? Did you realize you were incurring this debt, and if so, how did you deal with it? Please drop me a line and share your ideas.

Jerry Weinberg on Test Automation

I was recently part of a discussion on SHAPE about automated unit and functional testing. I was talking about buggy test code, and how frustrating (and sadly common) it can be to have automated test code that is unreliable. As a result, I try to keep test case code as simple and short as possible in an attempt to reduce bugs I might inadvertantly create. I do what I can to test the test code, but it can get out of control easily. It’s hard to unit test test code, and then I wonder why do I need tests for my test code? Isn’t that a smell that my test code is too complex?

I was musing that there is nothing more frustrating than buggy test code and buggy test harnesses, and Jerry agreed. He then pointed out something interesting. Jerry said:

There’s a kind of symmetry to all this, so that someone from Mars might not know, without being told, which was the product code and which was the test code. Each tests the other.

I had never thought of test code and its relationship to production code in this way before. This is a powerful and profound way of thinking about how we can “test our tests”, and is especially true with automated xUnit tests. It also strengthens the idea that automated test code needs to be treated as seriously as production code, and deserves the same kind of design, planning, resources and people behind it. Without symmetry, problems inevitably arise in test code especially.