Jonathan Kohl's Blog

Software Testing 2.0?

For so many years the Quality Assurance ideal has dominated software testing. "QA"-flavored software testing often feels like equal parts of Factory School and Quality School thrown together. When I was starting out as a tester, I quickly learned through hard experience that a lot of the popular software testing thought was built around folklore. I wanted results, and didn't like process police, so I often found myself at odds with the Quality Assurance community. I read writings by Cem Kaner, James Bach and Brian Marick, and worked on my testing skill.

When the Agile movement kicked into gear, I loved the Agile Manifesto and the values agile leaders were espousing, but the Agile Testing community rehashed a lot of the same old Factory School folklore. Instead of outsourcing testing to lesser-skilled, cheaper human testers, testing was often outsourced to automated testing tools. While there were some really cool ideas, "Agile Testing" ideals still frequently felt like testing didn't require skills, other than programming. I was frequently surprised at how "Agile Testing" thought was attracted to a lot of the old Factory School thoughts, like they were oppositely charged magnets. As a big proponent of skilled testing, I found I was often at odds with "Agile Testers", even though I agreed with the values and ideals behind the movement. Testing in that community did not always feel "agile" to me.

Then Test-Driven Development really got my attention. I worked with some talented developers who taught me a lot, and wanted to work with me. They told me they wanted me to work with them because I thought like they did. I was still a systems thinker, but I came at the project from a different angle. Instead of confirming that their code worked, I creatively thought of ideas to see if it might fail. They loved those ideas because it helped them build more robust solutions, and in turn, taught me a lot about testing through TDD. I learned that TDD doesn't have a lot to do with testing in the way I'm familiar with, but is still a testing school of thought. It is focused on the code-context, and I tend to do more testing from user contexts. Since I'm not a developer, and TDD is predominantly a design tool, I wasn't a good candidate for membership in the TDD community.

The Context-Driven Testing School is a small, influential community. The founders all had an enormous influence on my career as a tester. One thing this community has done is build up and teach skilled testing, and has influenced other communities. Everywhere I go, I meet smart, talented, thoughtful testers. In fact, I am meeting so many, that I believe a new community is springing up in testing. A community born of experience, pragmatism and skill. Testers with different skillsets and ideas are converging and sharing ideas. I find this exciting.

I'm meeting testers in all sorts of roles, and often the thoughtful, skilled ones aren't necessarily "QA" folks. For example, some of my thought-leader testing friends are developers who are influenced by TDD. Some skilled testers I meet are test automation experts, some are technical writers, some are skilled with applying exploratory testing concepts. All are smart, talented and have cool ideas. I am meeting more and more testers from around the world with different backgrounds and expertise who share a common bond of skill. I'm beginning to believe that a new wave of software testing is coming, and a new skills-focused software testing community is being formed through like-minded practitioners all over the world. This new community growing in the software development world is driven by skilled testers.

This is happening because skilled testers are sharing ideas. They are sharing their results by writing, speaking, and practicing skilled testing. Results mean something. Results build confidence in testers, and in the people who work with them. Skill prevails over process worship, methodology worship and tool worship. I've said before that skilled software testing seems to transcend the various methodologies and processes and add value on any software development project. I'm finding that other testers are finding this out as well. This new wave of skilled tester could be a powerful force.

Are you frustrated with the status quo of software testing? Are you tired of hearing the same hollow maxims like "automate all tests", "process improvement" and "best practices"? Do you feel like something is missing in the Quality Assurance and Agile communities when it comes to testing? Do you feel like you don't fit in a community because of your views on testing? You aren't alone. There are many others who are working on doing a better job than we have been doing for the past few years. Let's work together to push skilled software testing as far as it will go. Together, we are creating our own community of practice. The "second version" of software testing has begun to arrive.

Reckless Test Automation

The Agile movement has brought some positive practices to software development processes. I am a huge fan of frequent communication, of delivering working software iteratively, and strong customer involvement. Of course, before "Agile" became a movement, a self-congratulating community, and a fashionable term, there were companies following "agile" practices. Years ago I worked for a startup with a CEO who was obsessed with iterative development, frequent communication and customer involvement. The Open Source movement was an influence on us at that time, and today we have the Agile movement helping create a shared language and a community of practice. We certainly could have used principles from Scrum and XP back then.

This software business involves trade-offs though, and for all the good we can get from Agile methods, vocal members of the Agile community have done testing a great disservice by emphasizing some old testing folklore. One of these concepts is "automate all tests". (Some claimed agilists have the misguided gall to claim that manual testing is harmful to a project. Since when did humans stop using software?) Slavishly trying to reach this ideal often results in: Reckless Test Automation. Mandates of "all", "everything" and other universal qualifiers are ideals, and without careful, skillful implementation, can promote thoughtless behavior which can hinder goals and needlessly cost a lot of money.

To be fair, the Agile movement says nothing officially about test automation to my knowledge, and I am an enthusiastic supporter of the points of the Agile Manifesto. However, the "automate all tests" idea has been repeated so often and so loudly in the Agile community, I am starting to hear it being equated with so-called "Agile-Testing" as I work in industry. In fact, I am now starting to do work to help companies undo problems associated with over-automation. They find they are unhappy with results over time while trying to follow what they interpret as an "Agile Testing" ideal of "100% test automation".

The problems, like the positives of the Agile movement aren't really new. Before Agile was all the rage, I helped a company that had spent six years developing automated tests. They had bought the lie that vendors and consultants spouted: "automate all tests, and all your quality problems will be solved". In the end, they had three test cases developed, with an average of 18000 lines of code each, and no one knew what their intended purpose was, what they were supposed to be testing, but it was very bad if they failed. Trouble was, they failed a lot, but it took testers anywhere from 3-5 days to hand trace the code to track down failures. Excessive use of unrecorded random data sometimes made this impossible. (Note: random data generation can be an incredibly useful tool for testing, but like anything else, should be applied with thoughtfulness.) I talked with decision makers and executives, and the whole point of them buying a tool and implementing it was to help reduce the feedback loop. In the end, the tool greatly increased the testing feedback loop, and worse, the testers spent all of their time babysitting and maintaining a brittle, unreliable tool.

How did I help them address the slow testing feedback loop problem? Number one, I de-emphasized relying completely on test automation, and encouraged more manual, exploratory testing that was risk-based. This helped tighten up the feedback loop, and now that we had intelligence behind the tests, bug report numbers went through the roof. Next, we reduced the automation stack, and implemented new tests that were designed for quick feedback and lower maintenance. We used the tool to complement what the skilled human testers were doing. We were very strict about just what we automated. We asked a question: "What do we potentially gain by automating this test? And, more importantly, what do we lose?" The results? Feedback on builds was reduced from days to hours, and we had same-day reporting. We also had much better bug reports, and frankly, much better overall testing.

Fast-forward to the present time. I am still seeing thoughtless test automation, but this time under the "Agile Testing" banner. When I see reckless test automation on Agile teams, the behavior is the same, only the tools and ideals have changed. My suggestions to work towards solutions are the same: de-emphasize thoughtless test automation in favor of intelligent manual testing, and be smart about what we try to automate. Can a computer do this task better than a human? Can a human do it with results we are happier with? How can we harness the power of test automation to complement intelligent humans doing testing? Can we get test automation to help us meet overall goals instead of thoughtlessly trying to fullfill something a pundit says in a book or presentation or on a mailing list? Are our test automation efforts helping us save time, and helping us provide the team the feedback they need, or are they hindering us? We need to constantly measure the effectiveness of our automated tests against team and business goals, not "percentage of tests automated".

In one "Agile Testing" case, a testing team spent almost all of their time working on an automation effort. They had automated user acceptance tests, and were trying to automate all the manual regression tests to speed up releases. One release went out after the automated tests all passed, but it had a show-stopping, high profile bug that was an embarassment to the company. In spite of the automated tests passing, they couldn't spot something suspicious and explore the behavior of the application. In this case, the bug was so obvious, a half-way decent manual tester would have spotted it almost immediately. To get a computer to spot the problem through investigation would have required Artificial Intelligence, or a very complex fuzzy logic algorithm in the test automation suite, for one quick, simple, inexpensive, adaptive, yet powerful human test.

In another case, developers were so confident in their TDD-derived automated unit tests, they had literally gone for months without any functional testing, other than occasional acceptance tests by a customer representative. When I started working with them, they first defied me to find problems (in a joking way), and then were completely flabbergasted when my manual exploratory testing did find problems. They would point wide-eyed to the green bar in their IDE signifying that all their unit tests had passed. They were shocked that simple manual test scenarios could bring the application to its knees, and it took quite a while to get them to do some manual functional testing as well as their automated testing. When they did, I saw a marked improvement in the code they delivered once stories were completed.

In another "Agile Testing" case, the testing team had put enormous effort into automating regression tests and user acceptance tests. Before they were through, they had more lines of code in the test automation stack than what was in the product it was supposed to be testing. Guess what happened? The automation stack became buggy, unwieldly, unreliable, and displayed the same problems that any software development project suffers from. In this case, the automation was done by the least skilled programmers, with a much smaller staff than the development team. To counter this, we did more manual exploratory testing, and threw out buggy automation code that was regression test focussed. A lot of those tests should never have been attempted to be automated in that context. We found that the entire test environment had been optimized for the automated tests. The inherent system variablity the computers couldn't handle (but humans could!) had been attempted to be factored out. We did not have a system in place that was anything close to what any of our customers used. Scary.

After some rework on the testing process, we found it cheaper, faster and more effective to have humans do those tests, and we focussed more on leveraging the tool to help achieve the goals of the team. Instead of trying to automate the manual regression tests that were originally written for human testers, we relied on test automation to provide simulation. Running simulators and manual testing at the same time was a powerful investigative tool. Combining simulation with observant manual testing revealed false positives in some of the automated tests which had to been unwittingly released to production in the past.

Don't get me wrong - I'm a practitioner and supporter of test automation, but I am frustrated by reckless test automation. As Donald Norman reminds us, we can automate some human tasks with technology, but we lose something when we do. In the case of test automation, we lose thoughtful, flexible, adaptable, "agile" testing. In some tasks, the computer is a clear winner over manual testing. (Remember that the original "computers" were humans doing math - specifically calculations. Technology was used to automate computation because it is a task we weren't doing so well at. We created a machine to overcome our mistakes, but that machine is still not intelligent.)

Here's an example. On one application I worked on, it took close to two weeks to do manual credit card validation by testers. This work was error prone (we aren't that great at number crunching, and we tire doing repetitive tasks.) We wrote a simple automated test suite to do the validation, and it took about a half hour to run. We then complemented the automated test suite with manual exploratory testing. After an hour and a half of both automated testing (pure number crunching), and manual exploratory testing (usually scenario testing), we had a lot of confidence in what we were doing. We found this combination much more powerful than pure manual testing or pure automated testing. And it was faster than the old way as well.

When automating, look at what you gain by automating, and what you lose by automating a test. Remember, until computers become intelligent, we can't automate testing, only tasks related to testing. Also, as we move further away from the code context, it usually becomes more difficult to automate tests, and the trade-offs have greater implications. It's important to make considerations for automated test design to meet team goals, and to be aware of the potential for enormous maintenance costs in the long term. Please don't become reckless trying to fulfill an ideal of "100% test automation". Instead, find out what the goals of the company and the team are, and see how all the tools at your disposal can be harnessed to help meet those goals. "Test automation" is not a solution, but one of many tools we can use to help meet team goals. In the end, reckless test automation leads to feckless testing.