Category Archives: documentation

Content Pointer: Documenting Exploratory Testing

My latest testing article is on documentation and exploratory testing. You can read an online version here: Documenting Exploratory Testing. (If you don’t have a login, click the Preview button to read it.) You can order a hard copy from here.

I get this question frequently:

How do we document when we are exploratory testing?

This piece describes some of the things I do on testing projects.

Placating with Paper

Accountability. This is often a scary word for some people. Sometimes when we are fearful of the results of a decision, or of our work on a project, we will look at what we can do to offload responsibility. “At least I won’t be blamed if anything goes wrong.” might be the thinking. It’s hard to constantly offload onto other people though. After a while, we’ll run out of team members to blame, and we probably won’t be too popular. One easy way around this is to offload accountability onto an inanimate object: paper. Paper often comes in the form of specifications, design documents, test plans, test cases, or as I see more lately, proof of adherence to a popular development process. Paper might be substituted when we need to do something difficult, or provide reasons why a product isn’t working, or when we need to say something others may not want to hear. Paper with fancy words and jargon is often accepted as a short-term solution to project problems. Trouble is, what does it help in the long-term, when you are being paid to deliver software?

In one case, paper provided hope to end an unpleasant release. The development team had just come out of a meeting with our sales team and a couple of important clients. They were quite clear that we needed to deliver a particular feature in our next release. Unfortunately, it wasn’t working properly. Whenever we tested this new feature, the application crashed in a spectacular manner. There had been two attempts at redesigns, and everything else in the release seemed solid. The programmers were tired and demoralized, management didn’t want to hear about any problems anymore, and the testers were tired of testing and feeling the weight of the release resting on their shoulders. They felt singled out because whenever they logged a bug report, the rest of the team would accuse them of holding up the release. “Jonathan, why are we testing this feature in this way?” the test lead asked, and pointed to the specification document.

“We’re testing this because this is the most important feature in the release. Without this feature, our sales team is going to struggle, and our customers are going to be upset. Remember what we talked about in the meeting with both the customer and the sales team?” I asked. “Sure I do, but look at the spec. It doesn’t spell out what the customer asked for. If we test to the spec, we can ship! The tests will all pass if we only test according to the spec.”

The lead tester had found a loophole. The thinking was that if we tested only to the spec, and followed the document to the letter, we could pass the tests that were currently failing, and management would happily release the software. Everyone would be relieved, and furthermore the lead pointed out, who could blame the testers if anything went wrong? The design spec would be at fault, not the testers or developers. “But it flat-out doesn’t work!” I retorted. “Sure it does, according to the design spec.” they replied. “But the customer can’t use a design spec to do their jobs, can they? If our software doesn’t do what they expect it to do, are they supposed to print out the spec, clutch it to their chests and draw comfort and dollars from it? The spec is meaningless to the customer when they are trying to use our software. They need this feature to work in their context, to help them get their work done. They are relying on us to deliver software, not a piece of paper with a hollow excuse.”

I was a bit shocked at what this test lead proposed at first, but dealing with pressure in this way is not uncommon. A paper document can carry a lot of weight in an organization. People can draw a false sense of security from some official looking prose wrapped in corporate logos. I encouraged the tester to not fall for quick fix technicalities, especially if she wants to be a tester for very long. Fight the good fight, and while it may be hard in the short-term, the payoff in the long run is great. Happy customers, and the respect of your peers are just two payoffs. Being able to look at yourself in the mirror every day and know you have your integrity in tact is also something important, and no project deadline is worth sacrificing that for. A tester with a reputation for having no integrity will not be employable in the long-term.

On another project, paper was used in a different way. I was on one small team, and we met with other teams in the company periodically to share ideas. Another team came to a meeting and crowed about the “traceability” of documentation they were creating on a large project. We were on a small project at the time, and we were minimalists with documentation, preferring to deliver working software that could be deployed into production every two weeks. “That’s fine, but you don’t have traceability like we do!” they said. They bragged that for every method of code that would be written, there would be three documents generated as “project artifacts”.

They met with us again in a few months. We had delivered two projects that were helping the company make more money, and they were still talking about documents. “Can you show us the code?” we asked. “We’re still in the requirements-gathering phase”, they snapped. They were too busy to code, instead, they were writing documents — documents about the design, documents about a myriad of tools they could choose, and more documents documenting other documents. The company had spent several million dollars that year, and there wasn’t one line of code to show for it. They had so-called “traceability” though, with project documents, and committees reviewing documents, and committees reviewing other committees. An entire infrastructure to support this process was developed. There were literally teams of people working full-time churning out documents, but finding a programmer in all of this was not an easy task. Expensive tools to aid in paper generation were purchased, more people were hired to keep up, and project budgets were sacrificed on the altar of the great “project artifact” gods. (All stages of each properly documented and “signed off” on of course.)

Through all of this, management was placated with a blizzard of paper. “Do you have a demo yet?” they would ask, only to be inundated with mountains of UML diagrams, meeting minutes, decision matrices and vendor white papers. This satisfied management in the short-term. Look at all this paper – lots of work must be getting done.

In the end, paper didn’t deliver. There came a time when documents just didn’t cut it anymore. Management wanted a return on investment. They didn’t care about the development process du jour, detailed design documents, and requirements changes and budgets signed off in triplicate. They wanted to see the project working in production, helping them realize their business goals of a return to investors. There wasn’t much of a product there though – there were lots of documents, but precious little in the way of tested, delivered software.

The project went several years, and millions of dollars over budget. It was also a public embarrassment for the company. People identified with that project were blamed, and association with that project was an impediment to finding work with other companies who had heard about it through the grapevine.

What went wrong? The vendors, consultants and employees knew they could placate management with paper, and management didn’t demand a working solution early enough. This project could have done well, had they released working software incrementally, rather than churn out ever-growing mountains of paper. Instead of taking personal responsibility, the people working on the project placated customer demands with paper.

What were they thinking? Maybe they didn’t really know what they were doing. Maybe they hoped the programming elves would rescue them if they created enough documents. Maybe they got swept away in groupthink, and followed a process instead of working to deliver software. Maybe they collectively thought that a colleague would take responsibility. Maybe they just didn’t care. I don’t know.

The customer unwittingly paid people to generate a lot of paper, instead of getting them to write and demonstrate working software. They accepted paper designs instead of code that could be demonstrated. They accepted paper over skill, and they accepted paper instead of accountability and working software.

Accountability. Demand it from yourself, and from others. Placating with paper may get you through the short-term, but sooner or later you have to demonstrate your skill. Skill can be demonstrated individually as a programmer, as a tester, a technical writer, a project manager, and demonstrated collectively as a team. It will ultimately be demonstrated in your product. Does what you produce match your claims? If it doesn’t, a document and a nice explanation may get you out of a jam in the short-term, but in the long-term, you have to deliver what the customer needs. The customer won’t blame the document – they will blame you and your company, and move on to a competitor’s product. Where does that leave you?

Testing and Writing

I was recently involved on a project and noticed something interesting. I was writing documentation for a particular feature, and had a hard time expressing it for a user guide. We had made a particular design decision that made sense, and it worked intuitively but was difficult to write about coherently. As time went on, we discovered through user-feedback that this feature was confusing. I remembered how I had struggled to write about it, and shared this experience with my wife who worked as a technical writer for several years . She replied that this is very common. A lot of bugs that she and other writers found simply because a software feature was awkward to write about.

I’ve found that story cards in XP, or Scrum backlog items can suffer from incoherence, like any other written form. Incoherently written requirements, stories, use cases, etc. are frequently a sign that there is a problem in the design. There is something about actually writing down thoughts that helps us identify incoherence. When they are thoughts, or expressed verbally, we can defend problem spots with clarifications. When they are written down, we see them in a different light. What we think we understand may not be so clear when we capture our thoughts on paper.

Some of the most effective testers I’ve worked with were technical writers. They had a different perspective on the project than the rest of us because they had to understand the software and write about it in a way an end-user would understand. They find problems in the design when they struggle with writing about a feature. Hidden assumptions can come to light suddenly through capturing ideas on paper, and reading them. When testing, try writing down design ideas, as well as testing ideas, and see if what you write is coherent. Incoherent expressions of product features or test ideas can tell you a lot about development and testing.

Jennitta Andrea on Tests as Documentation

Jennitta Andrea sent me her position paper developed for the Tests as Documentation workshop. Unfortunately, she was unable to attend the workshop, but sent this paper with test examples. I like her approach, and encourage you to take a look at her examples. Here are some of her thoughts in the paper on the Tests as Documentation concept:

  • You must work VERY hard for your tests to be actually used as documentation for the system requirements.
    • They must be both organized and named extremely well so that someone looking for the details on some aspect of the system can find them.
    • They must be written with the reader in mind … all of the well known test automation patterns describe these details.
  • The automated acceptance tests must be built upon a clear and concise domain specific testing language (this is a superset of the ubiquitous language described by Evans).
  • The system requirements are spread out amongst a number of different kinds of tests
    • User acceptance tests cover the big-picture work flow and business process description
    • Unit tests cover the specific business rules for individual / small groups of components.
  • I still think you need some actual textual / graphical documentation that serves as an overview of the business process and the system. We recently put this kind of thing on a wiki at the very end of the project for the support team (should have done it incrementally during the project)
  • Stories should be thrown away once they are implemented. They should not be used as a permanent artifact for describing the system. A story is only a small chunk of the big picture, and it may contradict/overturn an earlier story. We need to create the big picture as it evolves when we implement various stories.

It’s interesting to note her emphasis on writing tests with an audience in mind which was a major theme in the workshop. She also noted that user acceptance tests document the business side of the product while the unit tests document the system process side of the product. This was also noted by one of the groups in the workshop.

I especially like the approach she has used for tabular test data. Check out “Example 2: Tabular Test Specification” of her test examples in the paper. The format for the test data in a spreadsheet is very intuitive and reads nicely.

Tests as Documentation Workshop Notes

At this year’s XP Agile Universe conference, Brian Marick and I co-hosted a workshop on Tests as Documentation. The underlying theme was: How are tests like documentation, and how can we use tests as project documentation?. Can we leverage tests to use as project documentation to help minimize wasteful documentation?

Since the most up to date information about a product is in the source code itself, how do we translate that into project documentation? In the absence of a tool to traverse the code, translate it and generate documentation, are tests a good place to look? Can we just take the tests we have and use them as documentation, or do we need to design tests a specific way?

We solicited tests from workshop participants, and had some sample tests developed in JUnit with the corresponding Java code, tests developed with test::unit and the corresponding Ruby code, and some FIT tests.
Brian organized the workshop in a format similar to Patterns or Writers workshops. This was done to facilitate interaction and to generate many ideas in a constructive way. Groups divided up to look at the tests, and to try to answer the questions from the workshop description. Once the pairs and groups had worked through these questions, they shared their own questions with the group. Here is a summary of some of the questions that were raised:

  1. Should a test be written so that it is understood by a competent practitioner? (Much like skills required to read a requirements document.)
  2. How should customer test documentation differ from unit test documentation?
  3. With regards to programmer tests: Is it a failure if a reader needs to look at the source code in order to understand the tests?
  4. What is a good test suite size?
  5. How do we write tests with an audience in mind?
  6. What is it that tests document?
  7. How should you order tests?
  8. Should project teams establish a common format for the documentation regardless of test type?
  9. How do we document exception cases?

Some of these questions were taken by groups, but not all of them. I encourage anyone who is interested to look at examples that might answer some of them and share them with the community. While discussion within groups and with the room as a whole didn’t provide a lot in the way of answers to these questions, the questions themselves are helpful for the community to think about tests as documentation.

Of the ideas shared, there were some clear standouts for me. These involve considering the reader – something I must admit I haven’t spent enough time thinking of.

The Audience

Brian pointed out an important consideration. When writing any kind of documentation whether it is an article, a book, project documentation etc., one needs to write with an audience in mind. When reviewing tests (and as a test writer myself), I notice that I don’t always write tests with an audience in mind. Often I’m thinking more about the test design, than the audience who might be reading the tests. This is an important distinction that we need to think about when writing tests if we want them to be used as documentation. Can we write tests with an audience in mind and still have them as effective tests? Will writing tests with an audience in mind help us write better tests? If we don’t write with an audience in mind, they won’t work very well as documentation.

What are We Trying to Say?

Another standout for me was what is it that tests document? We were fortunate to have example tests for people to review. The FIT tests seemed to be easier for non-developers to read, while the developers jumped into the Ruby test::unit and JUnit tests immediately. Some testers who weren’t programmers paired with developers who explained how to read the tests and what the tests were doing. I enjoyed seeing this kind of collaboration, and it got me thinking. More on that later. The point is, if we are writing a document, we need to have something to say. I’m reminded of high school English classes and learning how to develop a good thesis statement, and my teachers telling us we need to find something to say.

Order is Important

Another important point that emerged about tests as documentation was the order of the tests. Thinking of tests as documentation means thinking of the order of the tests not unlike chapters in a book, or paragraphs in a paper. A logical order is important. Without it we can’t get our ideas across clearly to the reader. It is difficult to read something that has jumbled ideas and doesn’t have a consistent, orderly flow.

With regards to the audience, one group identified two different potential audiences among programmers: designers and maintainers. A designer will need a different set of tests than a maintainer. Furthermore, the order of the tests developed will differ if one is a maintainer than if one is a designer. There are more audiences on the project than programmers, and these audiences may require a different order of tests.

Dealing with the “What is it that tests document?” question, one group felt that different kinds of tests document different things. For example, the unit tests the developers write will document the design requirements while the User Acceptance Tests will document the user requirements. The fact that some developers seemed more at home reading the unit tests, and some testers were more comfortable reading the FIT tests might give some credence to this. They are used to reading different project literature and might be more familiar with one mode over another.

Another important question was: “How do we define tests and explain what they are supposed to do?” If tests also should serve as project documentation and not just exercise the code or describe how to exercise the product in certain ways, the definition of tests will change according to how they are defined for a project.

Workshop Conclusions

I’m not sure we developed any firm conclusions from the workshop, though the group generated many excellent ideas. A workshop goal was to look at areas for further study, so we certainly met that. One idea that came up that I’ve been thinking about for a few months is to have meta descriptions in the automated tests that are more verbose. The tests would have program-describing details within the comments. A tool such as JavaDoc or RDoc could be used to generate project documentation from the specially tagged automated test comments. I like this idea, but the maintenance problem is still there. It’s easy for the comments to get out of date, and requires duplication of effort.

Most important to me were the questions raised about the audience, and how to write tests with an audience in mind. It appears that the tests we seem to be writing to date may not necessarily be taken on their own and used as documentation like requirements documents. None of the tests that we looked at sufficiently explained the product. The readers either had to consult the developer or look at the source code. This wasn’t a weakness or shortcoming of the tests, but showed us that tests as documentation is an area that needs more thought and work.

A couple of other very interesting observations were made. One was by a developer who said that you can tell whether tests were generated by Test Driven Development (TDD) or not by reading them. Another idea was that if one is reading tests and has to consult the source code to figure out what the program is doing might be a testing smell. These observations coupled with the tester/developer collaboration when reading tests got me thinking in a different direction.

My Thoughts

At the end of the workshop, I found myself less interested in tests serving as documentation to replace requirements documents, project briefs or other project documents. Instead, I started thinking about reading tests as a kind of testing technique. I started to imagine a kind of “literary criticism” technique to use to test our tests. This is an area that is hard to deal with. How thorough is our test coverage? Are our tests good enough? Are we missing anything? How do we know if our tests are doing the job they could be? I see a lot of potential to test our tests by borrowing from literary criticism.

Brian spoke about writer’s workshops as a safe place for writers to have practitioners, peers and colleagues look over each other’s work before they are published. This kind of atmosphere helps writers do better work and is a safe environment to get good constructive criticism before they are published and potentially savaged by the masses if they miss something important. For a “testing the tests” technique, instead of an us-versus-them relationship to simply negatively criticize, we could have test writers’ workshops to critique each other’s tests. The point is to have a safe environment to make the tests (and thereby the product) as solid as they could be before they are open to be potentially “…savaged by the masses,” for example, customers finding problems or faults of omission.

Here are three areas I saw in the workshop that could potentially help in testing the tests:

  1. I saw testers and developers collaborating, and it occurred to me that explaining what you have written (or coded) is one of the best ways of self-critiquing. When explaining how something works to someone else, I find myself noticing holes in my logic. Also, the other person also may spot holes in what has been written. That editor, or second set of eyes really helps as pair programming has demonstrated.
  2. I heard expert developers saying they could read *Unit tests and be able to tell immediately whether they were TDD tests or not. TDD tests are richer by nature they told us because they are more tightly coupled to the code. I thought that there is potential there for senior developers to read the tests to help critique constructively and find potential weak spots. One could have a developer outside of the pair that has been working read the tests as a type of test audit or editorial review.
  3. The emergence of a possible test smell: “If we have to look at the code to explain the program, are we missing a test?” prompted me to think of the potential for a catalog of test smells that reviewers could draw on. We look for bad “writing smells” using rules of grammar, spelling, etc. We could possibly develop something similar for using this style of review for our tests to complement the work that has already been done in the test automation area. This could involve reading the tests to find “grammatical” errors in the tests.

I still think there is a lot of potential to use tests as documentation, but it isn’t necessarily as simple as taking the tests we seem to be writing today and making them into project documentation in their original form. I encourage developers and testers to look at tests as documentation, and to think about how to use them to possibly replace wasteful documentation.

I learned a lot from the workshop, and it changed my thinking about tests as documentation. I’m personally thinking more about the “test the tests” idea than using tests as project documentation right now.

Communication and Documentation

One aspect of Agile Development that appeals to me, and need not only belong in the Agile Development school of thought, is the value that is placed on face-to-face communication. Paper doesn’t glue projects together; the people who are working on the project do. Collaboration works best when done face-to-face, and a written document cannot make the necessary decisions, or fully express the requirements, risks and mitigating factors of a project. I’m not against documentation, but I agree with James Bach, I’m against wasteful documentation. When the documentation distracts us from testing, we should evaluate what we are capturing in documents.

It seems that some successful development teams prefer “discussion with peers” over written documentation. Tim Van Tongeren posts about communication being more highly valued than documentation. Tim writes:

…the study showed that coordination and communication correlate with higher team performance and product quality

I am reminded of Principle 7 of the Context-Driven Testing School:

Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

We need to be careful and self-critical about what we do on our testing projects. Every action that does not directly contribute to our testing mission should be scrutinized even more than the actions that do.

Communication and collaboration facilitate decision making, and sharpening our skills with like-minded peers not only pushes us as individuals to perform, but benefits the team and the project as well. Writing project documents is difficult to do as a collaborative effort.

One point to think about is this: perhaps more accurate information about the project has already been captured in the program code and in the tests.

Tacit Knowledge and Underdetermination

Our perceptive equipment lags behind the conceptual skills that we as humans take for granted. As a consequence, linguistic explanations of concepts or feelings are represented by a limited medium. We often can’t provide a thorough linguistic representation of a concept or perceptual judgement to ourselves or to others. As a result, we frequently speak in metaphorical terms to describe feelings and concepts. If it is hard enough to express our own knowledge and thoughts linguistically, it is even harder to express the knowledge of others, especially in written documents.

Brian Marick has an article on Tacit Knowledge that underscores the underdetermination of project documents. The documents are an expression of the ideas and knowledge of experts, but in and of themselves do not provide the complete picture. I particularly liked the paragraph where he describes collaborating to gain tacit knowledge, and looking at a requirements document:

…as a tool for aligning the unexpressed wishes of the experts with the emerging abilities of the product