Brought to you by:
©2011 Kohl Concepts Inc.
I've written before about pairing with developers during Test-Driven Development(TDD), and I've been fortunate to work with very talented TDD developers who are apt to teach. I've learned a lot, and decided to try TDD in a programming role. Recently, I've taken off my tester hat and started doing TDD myself with test automation projects. I'm not completely there yet - I often need to do an architectural spike first when I'm developing something new. Once I have figured out a general design, or have learned how a particular library works, I throw away the spike code and start off development by writing a test. I then write enough code to get the test to pass, write a new test, add new code and repeat until the design is where I need it to be.
So what does this gain in test automation projects? I'm loathe to have test cases that are so complex that they themselves require testing. If our test cases are so complex that they are causing problems themselves, that's a test design smell. However, there are other kinds of software in our automation projects than just the test cases. In automation frameworks, we need special libraries, or adaptors, or a way to access an application we need to test, and all sorts of utilities that help us with automation. Since these utilities are still software, they are subject to the same problems that any other software development effort is. Sometimes there is nothing more frustrating than buggy test code, so we need to do what we can to make it as reliable as possible.
In my own development, I'm finding a lot of benefits doing TDD. My designs improve, because if they aren't testable, I know there is a problem. When I make my code testable, it suddenly becomes more usable and more reliable. Too often testers aren't given time to refactor test code. I've found that refactoring usually starts when technical debt in a test harness and custom test library are starting to interfere with productivity. When there are no unit tests for custom test library code, it can involve a few minutes to change the code, and several hours testing it. Having a safety net of unit tests helps immensely with refactoring. You can refactor your test code with greater confidence, and when it's done consistently with automated unit tests, with much greater speed. It just becomes a normal part of development.
Recently, I've found several bugs in my test library code in the elaborative phase of TDD that I didn't find testing manually. The opposite is also true, I tested libraries I was developing manually and found a couple of bugs that my unit tests didn't uncover. The TDD-discovered bugs required design changes that helped my design immensely. The tests guided the design into something different (and better) than what I had in my head when I started. However, after a couple of days of only running the unit tests to satisfy the "green bar", I found a big hole that was only uncovered by using the library the way an end user would. The balance of testing techniques is helpful. I have also adopted a practice of pairing a positive test with a negative test, a technique I learned from John Kordyback. If I do an assert_equal for example, I also do an assert_not_equal (using test::unit style). This has really come in handy at times where one assertion would work, but the other would fail.
TDD may not be for everyone, but I find it is a nice complement to other kinds of testing we can do. For my own work, it seems to suit the way I think about development. Even if the test cases I develop while programming are trivial and small, there is strength in numbers, and writing and using them helps assuage that tester voice in the back of my head that comes out when programming. I encourage other conventional testers who work on automation projects to give it a try. You may find that your designs improve, and you have a safety net of automated tests to give you more confidence in your automation code, especially when you need to enhance it down the road. At the very least, it helps you gain an appreciation and helps you communicate when working with TDD folks on a project.
I get asked a lot about Exploratory Testing on agile projects. In my position paper for the Canadian Agile Network Workshop last month, I described Exploratory Testing using personas. I'm reposting it here to share some of my own experience Exploratory Testing on agile projects.
Usability Testing: Exploratory Testing using Personas
Usability tests are almost impossible to automate. Part of the reason might be that usability is a very subjective thing. Another reason is that automated tests do not run within a business context. Program usability can be difficult to test at all, but working regularly with many end users can help. We usually don't have the luxury of many users testing full-time on projects; usually there is one customer representative. Once we get information from our test users, how do we continue testing when they aren't around? One possible method is Exploratory Testing with personas.
I've noticed a pattern on some agile teams. At first the customer and those testing have some usability concerns. After a while, the usability issues seem to go away. Is this because usability has improved, or has the team become too close to the project to test objectively? On one project, the sponsor rotated the customer representative due to scheduling issues. We were concerned at first, but a benefit emerged. Whenever a new customer was brought into the team, they found usability issues when they started testing. Often, the usability concerns were the same as what had been brought up by the testers and the customer earlier on, but had been contentious issues that the team wasn't able to resolve.
On another project using XP and Scrum, a usability consultant was brought in. They did some prototyping and brought in a group of end users to try out their ideas. Any areas the users struggled with were addressed in the prototypes. The users were also asked a variety of questions about how they used the software, and their level of computer skills, which we used to create user profiles or personas. As the developers added more functionality in each iteration, testers simulated the absent end users by Exploratory Testing with personas to more effectively test the application for usability. The team wanted to automate these tests, but could not. Exploratory Testing was much more effective at providing rapid feedback because it relies on skilled, brain-engaged testing within a context. The personas helped provide knowledge of the business context, and the way end-users interacted with the program in their absence. The customer representative working on the team also took part in these tests.
Tension on usability issues seemed to be reduced as well. These issues were no longer mere opinions. Now the team had something quantifiable to back up usability concerns. Instead of having differing opinions from developers, testers could say: "when testing with the persona 'Mary', we found this issue." This proved to be effective at reducing usability debates. The team compromised with most issues being addressed, and others not. There were still three contentious issues that were outstanding when the project had completed the UI changes. We scheduled time to revisit end-users and had some surprising results.
Each end-user struggled with the three contentious usability issues the testers had discovered, which justified the approach, but there were three more areas we had completely missed. We realized that the users were using the software in a way we hadn't intended. There was a flaw in our data gathering. Our first sample of users tested in our office, not their own. We had them work with the software at their own desks, and within their business context. Lesson learned: get the customer data when they are using the software in their own work environment.
On this project, Exploratory Testing with personas proved to be an effective way to compensate for limited full-time end user testing on the project. It also helped to provide rapid feedback in an area that automated tests couldn't address. It didn't replace the customer input, but worked well as a complementary testing technique with automation and customer acceptance testing. It helped to retain their voice in the usability of the product throughout development instead of sporadically, and helped to combat group think.
I've posted my Repeating the Unrepeatable Bug article that was published in Better Software magazine last month.
Michael Hunter has a good blog post that also deals with this issue. I'm glad that there are others who believe that there is no such thing as an "unrepeatable bug".
Brought to you by:
©2011 Kohl Concepts Inc.