Category Archives: bugs

bugs, politics, testing

I’m Finding Bugs, But It’s Pissing People Off. What Am I Doing Wrong?

A rookie tester asked me this question. They are experienced in software development, and their grad school work was in a highly specialized area of mathematics. They have been hired to do a very particular type of testing that they are uniquely qualified for. After getting over the mindset shift required to be an effective tester, they were pleased that they were finding bugs that they’d been hired to find. Then a wrinkle appeared. The very people who hired them to find the bugs got angry when they found the bugs they were being paid to find. Sound strange? It might, but it’s actually a common response to effective testing.
Here are some of my clarifying questions:

  • Are the bugs you are finding important, or trivial? Stakeholders on teams can get irritated if you inundate them with only trivial bugs.
  • How are your bug reports? Are they thorough? Do you provide enough information for devs to accurately reproduce the bugs?Programmers get irritated with bug reports they can’t reproduce and track down.
  • How’s your attitude? Do you laugh with glee at the misfortune of the dev team? Are you empathetic, or a condescending jerk? Is your language accusatory. blaming or condescending in any way?No one wants to be around someone who enjoys the schadenfreude.

No, the devs like my bug reports, and I’ve been as low-key and empathetic as possible. These are serious bugs that have probably been in the application since it was released several years ago. I’m tempted to stop logging the kinds of bugs that cause them to get mad at me.

Ok, you’re probably not doing anything wrong. In fact, you are probably doing something right. Don’t stop! You aren’t failing in your work, you are succeeding.

Since they seem to be doing some of the right things and aren’t knowingly or obviously antagonizing stakeholders, I assume that they really are finding important bugs, and that reaction is one telltale sign that they are being effective as a tester. I also encouraged them not to give up, but to do as Peter Block would recommend: move towards the resistance. That is a heuristic that tells us we are doing our jobs, and highlighting the really important problems. The resistance to being confronted with difficult problems that aren’t trivial to solve is just human nature.

As testers, if we are finding good problems, and we’re working with the team to help them, moving away from the resistance is a sure-fire way to relieve the pressure in the short-term, but lose credibility in the long term. If stakeholders realize we can’t be trusted, we’ve lost out ability to effectively test, observe information, and provide information that is useful for them.

I have a personal rule: if I am being pressured not to talk about something, or I feel like I should avoid a contentious issue, that means I absolutely must deal with it and talk about it. I try to be empathetic, understanding, use the right kind of language and not be a jerk, but I bring it up. Every time I do, something important is brought to the attention of the people who need to hear it. They may not like it, and they may not like to see me coming, but they know that I will always tell them the truth.
Early in my career, I was told not to bring up issues that were met with resistance. I was told that was a “career-limiting move.” My career is a testament to the opposite. Whenever I have faced resistance, stuck to my integrity and ethics, and talked about the hard problems the team had been taught to ignore, it has been a career-catapulting move.

So testers, if your work is pissing people off, the problems you are observing and reporting are important, and you aren’t being a jerk, don’t give up. It may hurt to point them out in the short-term, but it pays off handsomely in the long-term.

*Note: This doesn’t just apply to testing work, but any type of work that requires pointing out and helping solve problems. Whatever it is you do, don’t be discouraged if you are working on real problems and people are behaving strangely. Use that resistance to tell you that you are doing something right. The worst thing to do is give up and become silent.

bug reporting, bugs, pair testing, test cases, testing

Virtual Bugs

Mark McSweeny and I were talking about some of the challenges facing conventional testers on agile projects. One such challenge is what to do with bugs found during development, such as during Test-Driven Development when a tester and developer are pairing. It doesn’t seem fair to the developer to formally log bugs on a story before they have completed development on it. Many of them will be moot once the story is done, but some of them might squeak through. How do we keep track of them in a constructive way?

When they are pairing and developing tests, new test cases are added as they are generated, and the code is added to make them pass. However, sometimes when bugs are found during story development, the tester can overwhelm the developer and impede development progress. At this point, the developer and tester pair can split up, and the develoepr can pair with another developer and work towards completing the story. As a tester, what do I do with the bugs we discovered but couldn’t get the unit tests finished for?

On small teams, I will keep a running tab in my notes of these bugs. When the story is complete, I check my notes and test these scenarios first, and then log the ones that weren’t fixed during the story as “bug stories”. This is fine if there are a small number of developers, and I’m the only tester. This doesn’t scale well though. On slightly larger teams, I have also used a wiki to record these bugs which the other testers also reviewed and used. When they tested a story when it was complete, they would check the wiki first for any of these bugs. Any of these that weren’t addressed in development were then logged as bug stories or in a fault-tracking system. This can create classes of bugs which can create problems. It was hard to maintain the two systems, the wiki and the bug tracker.

As I was describing some of the problems I’ve come across with bugs found during story development, Mark cut through my usual verbosity with clarity, and said I was describing: “virtual bugs”. This is a lot more concise than my five minute hand waving explanation of this different class of bugs.

I have started calling the bugs found during story development “virtual bugs”. My question to other conventional testers on agile projects is: “How do you deal with virtual bugs”? Please share your experiences.

bug hunting, bugs, intermittence, unrepeatable bugs

Superbugs

My wife and I have friends and family in the health-care profession who tell us about “superbugs” – bacteria which are resistant to antibiotics. In spite of all the precautions, new technology and the enormous efforts of health care professionals, bugs still manage to mutate and respond to the environment they are in and still pose a threat to human health. In software development projects, I have encountered bugs that at least on the surface appear to exhibit this “superbug” behavior.

Development environments that utilize test-driven development, automated unit testing tools and other test-infected development techniques, in my experience, tend to generate very robust applications. When I see how much the developers are testing, and how good the tests are, I wonder if I’ll be able to find any bugs in their code at all. I do find bugs (sometimes to my surprise), but it can be much harder than in traditional development environments. Gone are the easy bugs an experienced tester can find in minutes in a newly developed application component. These include bounds conditions tests, integration tests and others that may not be what first come to mind to a developer testing their own code. However, in a test-infected development environment, most of these have already been thought of, and tested for by developers. As a tester, I have to get creative and inventive to find bugs in code that has already been thoroughly tested by the developers.

In some cases, I have collaborated with the developer to help in their unit test development efforts. <shameless plug> I talk about this more in next month’s edition of Better Software. </shameless plug> The resulting code is very hard for me to find bugs in. Sometimes to find any bugs at all, I have to collaborate with the developer to generate new testing ideas based on their knowledge of interactions in the code itself. The bugs that are found in these efforts are often tricky, time consuming and difficult to replicate. Nailing down the cause of these bugs often requires testers and developers pair testing. These bugs are not only hard to find, they are often difficult to fix, and seem to be resistant to the development efforts which are so successful in catching many bugs during the coding process. I’ve started calling these bugs “superbugs”.

It may be the case that certain bugs are resistant to the developer testing techniques, but I’m not sure if this is the case or not. I’ve thought until recently that these bugs also exist in traditionally developed code, but since testers spend so much time dealing with the bugs that test-infected development techniques tend to catch, they don’t have the time in the life of the project to find these types of bugs as frequently. Similarily, since they are difficult to replicate, they may not get reported as much by actual users, or several users may report the same problem in the form of several “unrepeatable” bugs.

Another reason *I* find them difficult to find might be due to my own testing habits and rules of thumb, particularly if the developer and I are working together quite closely. When we test together, I teach the developer some of my techniques, and they teach me theirs. When I finally test the code, both of our usual techniques have been tested quite well in development. Now I’m left with usability problems, some integration bugs that the unit testing doesn’t catch, and these so-called “superbugs”. Maybe the superbugs aren’t superbugs at all. Another tester might think of them as regular bugs, and may find them much more easily than I can because of their own toolkit of testing techniques and rules of thumb.

This behavior intrigues me none the less. Are we now able to find bugs that we didn’t have the time to find before, or are we now having to work harder as testers and push the bounds of our knowledge to find bugs in thoroughly developer-tested code? Or is it possible that our test-infected development efforts have resulted in a new strain of bugs?