Software Testing is a Game

David McFadzean and I wrote an article for Better Software magazine called The Software Development Game, which was published in the September/October edition. We describe applying game-like structures and approaches to software development teams. I’ve been asked for another article on applying game-like approaches to testing, so look for more in this space.

In the meantime, here are some of my current thoughts.

How is software testing like a game? If we think of software development as a game, or a situation involving cooperation and conflict with different different actors with different motivations and goals, software testing fits as a game within the larger software development game (SDG). While the SDG focuses more on policy, practices and determining an ideal mix of process, tools and technology, software testing is often an individual pursuit within a larger game format. There are two distinct styles of playing that game today: scripted testing and exploratory testing.

In the scripted testing game, we are governed by a test plan and a test case management system. We are rewarded for going through a management system and repeating test scripts and marking them as passed or failed. We are measured on our coverage of the tests that are stored in this test case management system. It is very important to stakeholders that we get as close to 100% coverage of the tests in that system as possible. This is a repetitive task, in a highly constricted environment. Anything that threatens the test team goal of high coverage of the tests within a test case management system tends to be discouraged, sometimes just implicitly. The metrics are what matter the most, so any activity, even more and better testing will be discouraged if it takes away from reaching that objective. “You, tester are rewarded on how many test cases you follow and mark as pass or fail in the system. If there is time after we get this number, you may look at other testing activities.”

In the exploratory testing game, testers are given some degree of self-determination. In fact, the latest version of the definition of exploratory testing emphasizes this:
“a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

The game player in exploratory testing is rewarded on the quality of their work, their approach, and the quality of the information that they provide. Exploratory testing works by being adaptable and relevant, so traditional ideas of metrics are often downplayed in favor of qualitative information. If a tester changes their mind when they are working on an assignment, and they have good reason to do so, and can make their case defensible, they are rewarded for changing things up because it helps make the product and our approach better. Coverage is important, but testers are rewarded on using multiple models of coverage (and discovering important new ones) rather than following the regression tests and counting that as complete coverage.

When I think about exploratory testing, I am reminded of the game Nomic where changing the rules is considered a valid move. Instead of a group effort like Nomic, a tester has the freedom to change their own course, and to help improve the approach of the overall team by demonstrating excellent work. “You, tester are rewarded by excellent testing, and your skill and ability to find and report important problems. We don’t care if you work through one test, or a thousand tests to get that information. The information you provide and how it improves the quality and value of our products is what we value.” It is deeply important for exploratory testers to be able to adapt to changing conditions, to uncover important information to help the team create value with their product, and for many, to get endorsement and the respect of their peers.

Now think of game play activities. Both scripted and exploratory testing approaches involve repetitive work. In the gamification space, we can explore these activities, and how we can apply game concepts to enhance the work. One area we can explore are video games. Imagine we are are playing a massively multiplayer online role-playing game (MMORPG). Some players like to perform repeated tasks that are unrelated to individual and shared game objectives. They like to repeat tasks to unlock features related to their character, such as acquiring new outfits or accessories for their character.

Other players are very achievement goal oriented – they try to reach individual goals and gain more points and unlock more achievement-based features in the game, and learn that when they co-operate with others, that they can achieve even more. One player type gets rewarded more for repeated individual tasks, while the other gets rewarded for trying to create achievement or quest value for themselves and others. The two types blend, as any player will have a subgoal of changing the appearance of my character, or acquiring accessories or currency to enhance quest or other achievement game play. People who are less adventurous and are more caught up in acquiring personal character attributes will also take part in quest events.

If you play these games and try to collaborate, you will often find other players who are at different points on this spectrum. It can be frustrating when your team members are more interested in building up their own character’s numbers than in helping the team as a whole. For example, if you are off on a raid, and one or more team members constantly stop to pick up health or currency points, your team will be let down and may have to regroup. It can be difficult to work together, since the perceived value of the game and the reward structures don’t match. One group want personal metrics and accessories, while the other care about non-character pursuits and the respect of their peers more.

In the software testing game, it is difficult to try to play an exploratory testing game in a system that is rigid, restrictive, and rewards testers for playing a scripted game. Conversely, those who are used to being rewarded on acquiring coverage counts in test case management systems can be very threatened when they are now rewarded on the quality of their testing skills and the information they provide. Testing tradition and common practice tends to think of testing as one sort of game, of which a test case management system is central. What they fail to realize is that there are many ways of playing this game, not just one. I want to explore this other alternatives in more detail through a gaming lens.

Testers and test managers, there is a lot more to our world than scripted testing and test case management tools. They are part of an ancient game (from at least the 1970s), which is struggling and showing its age in the face of new technology and new approaches to work that mobile technology and other styles of work are bringing. Modern testers work hard to play a game that fits the world around us. Are you measuring and rewarding them for playing the right kind of game, or the one you are most used to? Like the video game character who has gathered metrics and accessories, are you gathering these at the expense of the rest of the team’s goals, or are these measures and rewards helping the entire software development team create a better product?

Watch for more in this space as I explore software testing from the perspective of gamification, game theory and other game activities more deeply.

14 thoughts on “Software Testing is a Game”

  1. Testing is just a mind model and its all about the practices that makes us the real player. So I am totally agreed with you.

    Because more we will get involve in testing task more we will start getting amusement in this.

    I will also support you statement that “Software testing is a game” because testing as a profession is getting a very good response and many agile and fertile minds are enriching this testing world with their part of testing model and these testing models are getting great success in term of time and money like Exploratory testing (James Bach and Dr Cem Kerner) and Rapid testing (James Bach and MIchel ).

  2. An interesting article. In some ways, I agree software testing is a game–a very serious game, with serious consequences if major bugs are undetected. Perhaps it would be wiser to say that those who enjoy games, or puzzles, make excellent testers. What do you think?

    Thanks.

    1. Thanks for the comment.

      When we explore and apply game-like structures and apply them to areas that aren’t thought of traditionally as games, we are just using a model, or a perspective to help gain insight. When I design an application and I use gamification theory, it is about the engagement of the people who will use the app. Games are an interesting model to help guide thinking, and there is a lot to learn from them, but like all models, they are imperfect. I don’t make any value judgments, it is just a tool for thinking, and in my case, to help create or highlight practices and tools that support great testing work. That said, tere are some fascinating psychological and anthropological aspects that we have developed, and games and our engagement and interest in them are a reflection of some very deep areas of our selves and our relationships with others and the world around us.

      However, you mention it is a serious game, and there are many other serious game structures that are used in military, in financial modeling and other fields and disciplines that use game theory to help make critical decisions. Professional sports are also very serious games with billions of dollars at stake. Game-like structures and behaviors are all around us, and gamification concepts are making huge impacts in areas like education as we speak. it is a growing area of research and discovery, and I have used gamification ideas when designing products, so now I am turning it on the practice of software testing. I have found I can create more engaging products using gaming tools, David McFadzean and I have had success with the Software Development Game, and colleagues of ours have applied the SDG to other areas and have profited from it. I decided to also focus on testing using this model to see what I discover.

      As I said, I don’t make value judgments about these sorts of things, and I would disagree that those who enjoy games or puzzles would necessarily make excellent testers. Like anything else, some do, some don’t. I have known and worked with fantastic testers who didn’t like games, and hated solving puzzles. That doesn’t mean that they might not work well in a game-like structure, but they may not choose to look at it that way. I also know some people who are great at games and solving puzzles who aren’t so great as testers. Rather than use game ideas as value judgments on the quality of work, it is useful for me as a designer, and as a process coach and trainer when I apply gamification and other game concepts to some types of activities. Testing is an interesting concept to explore from this perspective.

      I will expand on this idea further, and hopefully more of my thinking will become clear as I write more. Watch this space for more thoughts.

      Thanks for commenting! I hope you share more of your ideas as I write more on the topic.

      -Jonathan

  3. Hi Jonathan,

    As a QA, this article gave me an opportunity that think about how I executed QA and how can it will be done.

    I think, it’s better if this brilliant article could be spread widely and thus provide more people with opportunity that rethink about what they did.
    So, I would like to translate your article “Software testing is game” and “Software development game” into Korean and post on my blog and software team blog.
    Hopefully I can get the permission and QA in KOREA have the opportunity to encounter this good article.

    My blog( http://angel927.tistory.com ) and software testing team blog( http://noogabar.tistory.com ) is non-commercial operations, and software testing-related articles I’ve seen are carrying mainly by translating.

    Wait for your positive response.

    Thanks,
    JIN

    1. Hi Jin,

      Thanks for the kind words. I’m glad you found it helpful. I’ll ve writing more on this topic, so I hope you continue to get value from these ideas.

      You have my permission to translate the content, all I ask for is attribution.

      Thanks for your feedback,

      -Jonathan

  4. Unfortunately, there is far too much testing going on out there, where testers act as robots running through scripts. That is especially true at large system integrators or test centers of excellence in large IT organizations.

    Testing is a fun and a creative job, and needs to be valued as such. While I hate self promotion, we are a couple of buddies trying to jump start a testing community and tools for exploratory testing and our motto is fun.

    Thanks for the great post!

    1. Thanks. Nothing wrong with a bit of self-promotion, especially if you have a good cause. “Testing is a fun and a creative job, and needs to be valued as such.” I can get behind that. Good luck with your testing tool development.

  5. Hi Jonathan,

    Great post! Love the analogy, one of the best I’ve read. It describes so well the different mindsets of scripted testers (and why they can easily become ‘zombie’ testers) and exploratory testers.

    I’ll use this analogy next time I’m struggling to get across the value of exploring in testing!

    Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.