Jonathan Kohl's Blog

Traits of Good Testers

People frequently ask me what to look for in potential candidates when hiring testers. Michael Hunter "The Braidy Tester" has a great post on The Hallmarks of a Great Tester. He has some good thoughts in this post. I recommend checking it out.

I would add the traits honesty and integrity, as well as someone who has courage to the list. Testers seem to end up in situations where ethical concerns may arise. A tester needs to know what their ethics are, and to stand for them. As Lesson 227 of Lessons Learned in Software Testing says:

"You don't have to lie. You don't have to cover up problems. In fact, as a tester, the essence of your job is to expose problems and not cover them up. You don't have to, and you never should compromise your integrity. Your power comes from having the freedom to say what's so, to the people who need to hear it. If you compromise your reputation for integrity, you'll weaken one of the core supports of your power." *

Discovering these traits in a potential candidate can be difficult, but there are effective ways of interviewing to tell if someone is a great tester or not. Check out Johanna Rothman's Hiring Technical People blog for more information on hiring techies.

*Excerpted from p.200 of Lessons Learned in Software Testing.

Gerard Meszaros on Computer Assisted Testing

At last night's Extreme Tuesday Club meeting, Gerard Meszaros had a great way of describing what I do using automated scripts with Ruby combined with exploratory testing. (See my blog post about this here: Test Automation as a Testing Tool.)

As I was describing using a Ruby script to automate some parts of the application I'm testing, and then taking over manually with exploratory testing, Gerard said:

"Sounds like you are using your Ruby program to set up the test fixture that you will then use as a starting point for exploratory testing."

I hadn't thought of what I was doing in this way before, and I like the idea. It is a great way to succinctly explain how to complement the strengths of a testing tool and a brain-engaged tester.

Check out Gerard's Patterns of xUnit Test Automation site for more of his thoughts on testing.

Dehumanizing the User

As Mike Cohn points out in Advantages of User Stories for Requirements, writing requirements documents with the phrase: "the system shall" causes us to lose focus on the user. We aren't writing something to satisfy the system, we are writing something to satisfy the needs of the user, the customer.

Maybe this is another encapsulation endeavor. Is using this kind of language an implicit (or unconscious) attempt to hide the complexity of working with people into a simplified interface called "the system"? As we encapsulate the needs and wants of the user within this black box, is this system an accurate representation of the customer's original knowledge? Does it meet their needs? How can we tell?

What do we lose when we shift focus from being customer-obsessed to talking more in terms of the system? It's one thing to satisfy a system, and quite another to satisfy the customer. Language is very important, and if the "user" or "customer" are being talked about less on a project, there is a danger that we may think about the customer and their needs less. Furthermore, if we refer less to the user and more to the system we are developing when looking at requirements and needs, we dehumanize the user. The context of satisfying a requirement at the system level and at the customer level can be two very different things. Customers worry about how usable the software is, sometimes more so than the fact that the requirement exists in the system concept.

If we dehumanize the user, why shouldn't the user dehumanize us? Instead of a development relationship based on collaboration focused on meeting the needs of the customer, we run the risk of becoming commodities to each other. What's to stop the customer from treating us as a commodity if we treat them as one? If communication breaks down and we dehumanize each other, how can we meet the needs of the customer? How can they be sure their needs are met? Can we trust each other?

Even when we use methodologies that are based on collaborating with the customer, we should be careful not to force them into a position they aren't comfortable with. A key for overcoming technical bullying is effective communication. We need to treat the customer as a human being who has needs and feelings like we do, and communicate with them. With proper two-way communication, we can make sure they understand what we are doing, and enable them to ensure we are meeting their needs. Words are important, and listening to the customer is even more important. If we really listen, we can find out what the customer needs and develop that. The customer needs to be satisfied, not necessarily the system. Our language should reflect this.

Dehumanizing Software Testing

I was talking with Daniel Gackle, a software developer and philosophical thinker, about developing software and systems. Daniel mentioned in passing that we often forget about the humanity in the system - the motivation of the human who is using the tool, or doing the work. This comment resonated with me, particularly with regards to some software testing practices.

Sometimes when we design test plans, test cases, and use testing tools, we try to minimize the humanity of the tester. The feeling might be that humans are prone to error, and we want solid, repeatable tests, so we do things to try to minimize human error. One practice is the use of procedural test scripts. The motivation often is that if the test scripts are detailed enough, anyone can repeat them, with a benefit that there will be less chance for variability or error. The people who write these tests try to be as detailed as possible with one motivation being that the test must be repeated the same way each time. It is not desirable to have variation because we want this exact test case to be run the same way every time it is run.

This type of thinking spills over into test automation as well. There is a push to have tools that can be used by anyone. These tools take care of the details, all a user needs to do is learn the basics and point and click and the tool does the rest for us. We don't need specialists then, the tool will handle test case design, development and execution.

Without going into the drawbacks that both of these approaches entail, I want to focus on what I would call the "dehumanizing of software testing". When we place less value on the human doing the work, and try to minimize their direct interaction with software under test, what do we gain, and what do we lose?

In The Dumbing Down of Programming, Ellen Ullman describes some of what is lost when using tools that do more work for us. Ullman says:

"the desire to encapsulate complexity behind a simplified set of visual representations, the desire to make me resist opening that capsule -- is now in the tools I use to write programs for the system."

It is today, (as it was in 1998 when this article was written), also in procedural test cases (or scripts) and in the popular "capture/replay" or event recording testing tools. Both seek to "encapsulate complexity" behind an interface. Both practices I would argue, lead to dehumanizing software testing.

Ullman mentions that the original knowledge is something we give up when we encapsulate complexity:

"Over time, the only representation of the original knowledge becomes the code itself, which by now is something we can run but not exactly understand."

When we don't think about the human interaction that I believe is so valuable to software testing, we need to be aware of what we are giving up. There is a trade-off between having interactive, cognitive testing, and testing through interfaces where the complexity is hidden from us. There are cases where this is useful, we don't always need to understand the underlying complexity of an application when we are testing against an interface. However, as Ullman says, we should be aware of what is going on:

"Yet, when we allow complexity to be hidden and handled for us, we should at least notice what we're giving up. We risk becoming users of components, handlers of black boxes that don't open or don't seem worth opening."

When is hiding the complexity dehumanizing software testing, and when is it a valuable practice? I would argue that as soon as we encapsulate complexity to the point that the tester is discouraged from interacting with the software with their mind fully engaged, we are dehumanizing software testing. If a tester is merely following a script for most of their testing work, they are going to have trouble being completely mentally engaged. If a testing tool is trying to do the work of test design and execution, it will be difficult for the tester to be completely engaged with testing. I believe we miss out on their skills of observation, inquiry and discovery when we script what the tester needs to do and they do most of their work by rote.

The important thing to me is to understand what the trade-offs are before we adopt a testing practice. When we forget the humanity of the system, we will probably get results other than what we intended. We need to be aware of how people impact the use of tools, and how people work on projects, because after all, it's still the people who are key to project success.