Category Archives: context-driven testing

Software Testing 2.0?

For so many years the Quality Assurance ideal has dominated software testing. “QA”-flavored software testing often feels like equal parts of Factory School and Quality School thrown together. When I was starting out as a tester, I quickly learned through hard experience that a lot of the popular software testing thought was built around folklore. I wanted results, and didn’t like process police, so I often found myself at odds with the Quality Assurance community. I read writings by Cem Kaner, James Bach and Brian Marick, and worked on my testing skill.

When the Agile movement kicked into gear, I loved the Agile Manifesto and the values agile leaders were espousing, but the Agile Testing community rehashed a lot of the same old Factory School folklore. Instead of outsourcing testing to lesser-skilled, cheaper human testers, testing was often outsourced to automated testing tools. While there were some really cool ideas, “Agile Testing” ideals still frequently felt like testing didn’t require skills, other than programming. I was frequently surprised at how “Agile Testing” thought was attracted to a lot of the old Factory School thoughts, like they were oppositely charged magnets. As a big proponent of skilled testing, I found I was often at odds with “Agile Testers”, even though I agreed with the values and ideals behind the movement. Testing in that community did not always feel “agile” to me.

Then Test-Driven Development really got my attention. I worked with some talented developers who taught me a lot, and wanted to work with me. They told me they wanted me to work with them because I thought like they did. I was still a systems thinker, but I came at the project from a different angle. Instead of confirming that their code worked, I creatively thought of ideas to see if it might fail. They loved those ideas because it helped them build more robust solutions, and in turn, taught me a lot about testing through TDD. I learned that TDD doesn’t have a lot to do with testing in the way I’m familiar with, but is still a testing school of thought. It is focused on the code-context, and I tend to do more testing from user contexts. Since I’m not a developer, and TDD is predominantly a design tool, I wasn’t a good candidate for membership in the TDD community.

The Context-Driven Testing School is a small, influential community. The founders all had an enormous influence on my career as a tester. One thing this community has done is build up and teach skilled testing, and has influenced other communities. Everywhere I go, I meet smart, talented, thoughtful testers. In fact, I am meeting so many, that I believe a new community is springing up in testing. A community born of experience, pragmatism and skill. Testers with different skillsets and ideas are converging and sharing ideas. I find this exciting.

I’m meeting testers in all sorts of roles, and often the thoughtful, skilled ones aren’t necessarily “QA” folks. For example, some of my thought-leader testing friends are developers who are influenced by TDD. Some skilled testers I meet are test automation experts, some are technical writers, some are skilled with applying exploratory testing concepts. All are smart, talented and have cool ideas. I am meeting more and more testers from around the world with different backgrounds and expertise who share a common bond of skill. I’m beginning to believe that a new wave of software testing is coming, and a new skills-focused software testing community is being formed through like-minded practitioners all over the world. This new community growing in the software development world is driven by skilled testers.

This is happening because skilled testers are sharing ideas. They are sharing their results by writing, speaking, and practicing skilled testing. Results mean something. Results build confidence in testers, and in the people who work with them. Skill prevails over process worship, methodology worship and tool worship. I’ve said before that skilled software testing seems to transcend the various methodologies and processes and add value on any software development project. I’m finding that other testers are finding this out as well. This new wave of skilled tester could be a powerful force.

Are you frustrated with the status quo of software testing? Are you tired of hearing the same hollow maxims like “automate all tests”, “process improvement” and “best practices”? Do you feel like something is missing in the Quality Assurance and Agile communities when it comes to testing? Do you feel like you don’t fit in a community because of your views on testing? You aren’t alone. There are many others who are working on doing a better job than we have been doing for the past few years. Let’s work together to push skilled software testing as far as it will go. Together, we are creating our own community of practice. The “second version” of software testing has begun to arrive.

Consultants Camp Report

In September, I attended Consultants Camp. This was my first time there, and I wasn’t sure what to expect. I had the camp handbook, and read a post by Dale Emery describing Camp. (It’s worth a read, like all of Dale’s writings.)

This was about all I knew of Camp when I flew out of Calgary. Now that I’ve attended Camp, and have had time for reflection, I have something to share. Camp had a big effect on me, and I got a lot out of it. I came back home and Elizabeth said: “This has been good for you. You’ve come home rested, energized and full of new ideas.” I struggled with a good word to describe what Camp meant for me, and found an article called Renewalby fellow Camper Rick Brenner. “Renewal” describes what I felt after attending Camp. Rick says:

Just about every year I attend a conference called Consultants’ Camp. It’s a group of consultants, IT specialists and process experts who meet annually in Mt. Crested Butte, Colorado, for a week of self-organized collegiality and fun. In some ways, it’s a conference like any other — there’s a continental breakfast, days full of sessions, and there is a program. By the end of the conference many of us feel tired and full. Learning is everywhere.

In other ways Camp is unique. The setting, the Colorado Rockies, is inspirational. Attendees give all sessions. There is no sponsor. Every day, there’s a long break in mid-afternoon, which today I’m using to write this essay. Lunch isn’t provided, but most of us ante up and share soup and sandwiches and stimulating conversation. For me, and I think for all of us, there’s a feeling of belonging.

Renewal is a time to step out of the usual routine and re-energize. I feel good to be here, with these people — colleagues and friends. Renewal can be a large block of time, as Consultants’ Camp is, or it can be a few minutes. We find renewal in weekends, vacations, days off, even in a special evening or hour in the midst of routine.

I couldn’t have said it better myself.

Camp had some times of joy, and there were difficult times we experienced with sadness. I talked with other first-time campers, and a common theme emerged: Consultants Camp is a community that is not only highly intellectual, but also understands people’s humanity and emotion. This is a community that cares for each other.

One of my favourite parts of Consultants Camp was the opportunity to spend a lot of time with James Bach. James generously offered to spend time with me talking about the testing craft, and having me work through testing exercises. I learned a tremendous amount from the time we spent together, and I have renewed direction on improving my skills as a tester.

I’ve followed James since about 1999. I had been testing as an intern student for a few months, and was in a unique position of leadership. I found I was struggling to explain what I was doing when training other testers. I realized I was too focused on technology and I was drawing from other disciplines when testing that I hadn’t realized. Since I had studied Philosophy and Business as well as Technology, there were non-computer related influences I didn’t realize coming through in my thinking about testing. I had studied Inductive Logic in university, and found when I read James’ work, the correlation of software testing with philosophy I had been bumping up against was spelled out already for me. I was pleased to see James recommend books on Abductive Inference, and I started reading everything I could that he had written. What he was saying was matching my beliefs and experience, and he had a lot of great information for me to use.

James continues to be an influence because I greatly respect his commitment to teaching testing skills in the software testing field. Furthermore, my experiences show that what he says is true. We tend to think alike when it comes to testing, and I have the benefit of his knowledge and experience to draw on in my own work. I (and other testers) are indebted to his willingness to share great ideas, and teach what he knows.

I have used James’ Heuristic Risk-Based Testing strategy ideas a lot, particularly on agile teams. However, it wasn’t until I worked with James that I really began to own the concept of heuristics, and developingmnemonics to help remember and apply my own heuristics when testing.

I was pleased when I was finally able to meet James in person for the first time last year, and during Consultants Camp we picked up where we left off. James posed problems, I worked on solving them, and he continuously challenged me to push myself harder as a tester and as a thinker. I learned a lot, much like I have from other great (and very rare) teachers I’ve had who have challenged me this way: Dr. Michael Kubara, Dr. John Rutland, and my Father come to mind. We also spent time working on the motivations behind the Context-Driven Testing School principles, and talking about testing, working and life. I appreciate the time we were able to spend together, and have a lot of work to do and new areas to explore in my learning.

Conventional Testers on Agile Projects – Getting Started Continued

Some of what you find out about agile methods may sound familiar. In fact, many development projects have adopted solutions that some agile methods employ. You may have already adjusted to some agile practices as a conventional tester without realizing it. For example, before the term “agile” was formally adopted, I was on more traditional projects that had some “agile” elements:

  • when I started as a tester, I spent a lot of my first year pair testing with developers
  • during the dot com bubble, we adopted an iterative life cycle with rapid releases at least every two weeks
  • one project required quick builds, so the team developed something very similar to a continuous integration build system with heavy test automation
  • developers I worked with had been doing refactoring since the early ’80s. they didn’t call it by that name, and used checkpoints in their code instead of xUnit tests that would be used now
  • in a formal waterfall project, we had a customer representative on the team, and did quick iterations in between the formal signoffs from phase to phase
  • one project adapted Open Source-inspired practices and rapid prototyping

These actions were done by pragmatic, product-focused companies who needed to get something done to please the customer. Many of these projects would not consider themselves to be “agile” – they were just getting the job done. The difference between them and an agile development team is that the agile methods are a complete methodology driven towards a certain goal rather than a team who has adjusted some practices to improve what they are doing.

Other conventional testers tell me about projects they were on that were not agile, but did agile-like things. This shouldn’t be surprising. The iterative lifecycle has been around for many years (at least back to the 1940s). There are a lot of methodologies that people have used, but not necessarily codified into a formal method within the iterative lifecycle as some agile champions have. A lot of what agile methods talk about isn’t new. Jerry Weinberg has said that methods employed on the Mercury project team he was on in the early ’60s looks to be indistinguishable from what is now known as Extreme Programming.1

Another familiar aspect of agile methods is the way projects are managed. Much of the agile management theory draws very heavily from the quality movement, lean manufacturing, and what some might call Theory Y management. Like the quality pundits of past, many agile management writers are once again educating workers about the problems of Taylorism or Theory X management.

What is new with agile methods, are comprehensive methodology descriptions that are driven from experience. From these practices, discplined design and development methodologies have improved rapidly, such as Test-Driven Development. Most importantly, a shared language has emerged for practices like “unit testing”, “refactoring”, “continuous integration” and others – many of which might have been widely practiced but called different things. This shared language helps a community of practice share and improve ideas much more efficiently. Common goals are much more easily identified when everyone involved is using the same terminology. As a result, the needs of the community have been quickly addressed by tool makers, authors, consultants and practitioners.

This has several implications for conventional testers that require some adjustments:

  • a new vocabulary of practices, rituals, tools and roles
  • getting involved in testing from day one
  • testing in iterations which are often 2-4 weeks long
  • an absence of detailed, formalized requirements documents developed up front
  • requirements done by iteration in backlogs or on 3×5 story cards
  • often, a lack of a formal bug-tracking system
  • working knowledge of tools such as refactoring and TDD-based IDEs, xUnit automation and continuous integration build tools
  • a team focus over individual performance
  • developers who are obsessed with testing
  • working closely with the entire team in the same work area
  • not focusing on individual bug counts or lines of code
  • less emphasis on detailed test plans and scripted test cases
  • heavy emphasis on test automation using Open Source tools

Some of these changes sound shocking to a conventional tester. Without a detailed requirements document, how can we test? Why would a team not have a bug tracking database? What about my comprehensive test plans and detailed manual regression test suites? Where are the expensive capture/replay GUI automation tools? How can we keep up with testing when the project is moving so quickly?

A good place to address some of these questions is: Lessons Learned in Software Testing: A Context-Driven Approach by Cem Kaner, James Bach and Bret Pettichord.

We’ll address some of these challenges in this series, as well as examples of testing activities that conventional testers can engage in on agile projects.

1 p. 48 “Iterative and Incremental Development: A Brief History”, Larman and Basili, 2003

Communication and Documentation

One aspect of Agile Development that appeals to me, and need not only belong in the Agile Development school of thought, is the value that is placed on face-to-face communication. Paper doesn’t glue projects together; the people who are working on the project do. Collaboration works best when done face-to-face, and a written document cannot make the necessary decisions, or fully express the requirements, risks and mitigating factors of a project. I’m not against documentation, but I agree with James Bach, I’m against wasteful documentation. When the documentation distracts us from testing, we should evaluate what we are capturing in documents.

It seems that some successful development teams prefer “discussion with peers” over written documentation. Tim Van Tongeren posts about communication being more highly valued than documentation. Tim writes:

…the study showed that coordination and communication correlate with higher team performance and product quality

I am reminded of Principle 7 of the Context-Driven Testing School:

Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

We need to be careful and self-critical about what we do on our testing projects. Every action that does not directly contribute to our testing mission should be scrutinized even more than the actions that do.

Communication and collaboration facilitate decision making, and sharpening our skills with like-minded peers not only pushes us as individuals to perform, but benefits the team and the project as well. Writing project documents is difficult to do as a collaborative effort.

One point to think about is this: perhaps more accurate information about the project has already been captured in the program code and in the tests.