Jonathan Kohl's Blog

Testing and Writing

I was recently involved on a project and noticed something interesting. I was writing documentation for a particular feature, and had a hard time expressing it for a user guide. We had made a particular design decision that made sense, and it worked intuitively but was difficult to write about coherently. As time went on, we discovered through user-feedback that this feature was confusing. I remembered how I had struggled to write about it, and shared this experience with my wife who worked as a technical writer for several years . She replied that this is very common. A lot of bugs that she and other writers found simply because a software feature was awkward to write about.

I've found that story cards in XP, or Scrum backlog items can suffer from incoherence, like any other written form. Incoherently written requirements, stories, use cases, etc. are frequently a sign that there is a problem in the design. There is something about actually writing down thoughts that helps us identify incoherence. When they are thoughts, or expressed verbally, we can defend problem spots with clarifications. When they are written down, we see them in a different light. What we think we understand may not be so clear when we capture our thoughts on paper.

Some of the most effective testers I've worked with were technical writers. They had a different perspective on the project than the rest of us because they had to understand the software and write about it in a way an end-user would understand. They find problems in the design when they struggle with writing about a feature. Hidden assumptions can come to light suddenly through capturing ideas on paper, and reading them. When testing, try writing down design ideas, as well as testing ideas, and see if what you write is coherent. Incoherent expressions of product features or test ideas can tell you a lot about development and testing.

Remuneration and Punished by Rewards

Lately I've been reading Punished by Rewards by Alfie Kohn. He shows how incentives can work to get people to do things in the short-run, but ultimately fail. I frequently get asked how to measure tester's performance, and I am often at odds with people looking for hard numbers to measure and reward testers by. "Without numbers on _____, how can I do a performance evaluation? How can I use "pay for performance" unless I have hard numbers? I need to measure testers to motivate them." Alfie Kohn explains why these kinds of practices can backfire better than I can. If you are a manager, teacher, or leader of any sort with people reporting to you, I urge you to read this book. (If you aren't in a position of leadership, feel free to read it too.) Even if you disagree with it, at least it will provide a different perspective.

Questions about fees come up for anyone who charges for their services, and I was pleased that Alfie Kohn dealt with this issue in chapter 10 of Punished by Rewards. On p. 183:

"When someone contacts me about giving a lecture or writing an article, I ask how much money is involved and often negotiate for the maximum amount that seems to be fair and that the organization can afford to pay. Then, assuming we have come to an agreement, I do my best to not think about the money again. I do this because I fear the consequences of construing what I am doing in terms of what I am being paid: eventually I might find myself thinking, "A is paying me twice as much as B, so I'd better do twice as good a job for A." If I ever reach that point, my integrity will be gone, and my intrinsic motivation will have fled along with it."

Kohn goes on to explain that if we don't decouple the task from the compensation, we run the risk of being too focused on what we get at the end at the expense of what we are supposed to be working on right now. That "intrinsic motivation" is why we do the work we do, and why we do it well. If we sacrifice it by not being ethical, or allowing money to take precedence, our work will suffer.

Money, status, fear, or "gold stars" are not good motivators. Kohn answers the question about motivation on p. 181 of Punished by Rewards:

"...it is possible to get people to do something. That is what rewards, punishments and other instruments of control are all about. But the desire to do something, much less to do it well, simply cannot be imposed; in this sense, it is a mistake to talk about motivating other people. All we can do is set up certain conditions that will maximize the probability of their developing an interest in what they are doing and remove the conditions that function as constraints."

On p. 182, Kohn echos W. Edwards Deming: "Pay is not a motivator." Kohn says this about pay:

"Pay people generously and equitably. Do your best to make sure they don't feel exploited. Then do everything in your power to help them put money out of their minds."

This might sound foreign at first, but think about how groups of people who do extraordinary things are motivated. Look at Open Source software where legions of talented people work hard for no pay. Look at religious and political organizations where people believe in a cause, and are willing to commit to it and share their talents and time. Pay isn't a motivator for people who believe in something they feel is important. Their intrinsic motivation is powerful, and when they become disillusioned, they drift. Often, when people lose track of the purpose, it is replaced by the appearance of purpose, which can be dangerous.

Be careful what you measure, and what you reward based on those measurements. If that intrinsic motivation to do work is clouded by the rewards, we will get results that are surprising, and not necessarily in line with our goals as leaders. In fact, we may eventually frustrate our overall goals as they are superseded by measurement and rewards at the micro level. On p. 183, Kohn says:

"Providing feedback that employees can use to do a better job ought never to be confused or combined with controlling them by offering (or witholding) rewards."

He goes on to explain how this kind of action can have disastrous results. Those of us who lead testers need to be sure that the way we measure and reward does not distort or remove the tester's intrinsic motivation for testing. If the tester has no intrinsic motivation anymore, we run the risk of not getting the kind of work completed that the company hopes for.