Category Archives: tools

gamification, test quests, testing, testing debt, tools, UX

Designing a Gamification Productivity Tool

4 Comments

Gamification and Software Testing

I haven’t spoken about this project publicly because we never got to a public release. Software testing tools represent a tiny market, so they are incredibly difficult to fund. Some of you have asked me about gamification tools with testing, so I thought I would share this brain dump.

A few years ago, I was asked to help a development team that had significant regulatory issues, and frequently accrued testing debt. The product owner’s solution was to periodically have “testing sprints” where other team members helped the over burdened test teams catch up. There was just one problem: the developers HATED helping out with 2 weeks of testing, so I was asked to do what I could to help.

A couple of the senior architects at this company were very interested in Session Tester and asked me why I had put game mechanics in a testing tool. I didn’t really realize at the time I had put game mechanics in, I was just trying to make something useful and engaging for people. So I started talking with them more about game design, and they encouraged me to look into MMOs and co-operative games. The team played games together a great deal, so I learned about the games they enjoyed and tried to incorporate mechanics

I set up a game-influenced process to help structure testing for the developers, and taught them the basics of SBTM. They LOVED it, and started having fun little side contests to try to find bugs in each other’s code. In fact, they were enjoying testing so much, they would complain about having to go back to coding to fix bugs. They didn’t want to do it full time, but a two week testing sprint under a gamified, co-operative model with some structure (and no horrible boring test cases) really did the trick.

Eventually, I worked with some of the team members with a side-project, and the team lead proposed creating a tool to capture what I had implemented. This was actually extremely difficult.  We started with what had been done with Session Tester, and went far beyond that, looking at a full stack testing productivity tool. One of the key aspects of our approach that differed from the traditional ET and scripted testing approaches was the test quest. As I was designing this test tool, I stumbled on Jane McGonigal’s work and found it really inspiring. She was also a big proponent of the quest as a model for getting things done in the real world. Also, we were very careful in how we measured testing progress. Bug counts are easily gamed and have a lot of chance. I have worked in departments that measured on bug counts in the past, and they are depressing if you are working on a mature product while your coworkers are working on a buggy version 1.0.

One thing Cem Kaner taught me was to reward testers based on approach rather than easily counted results, because they can’t control how many bugs there may or may not be in a system. So we set up a system around test quests. Also, many people find pure exploratory testing (ET) too free form and it doesn’t provide a sense of completion the way scripted test case management tools do. And when you are in a regulatory environment, you can’t do ET all the time, and test cases are too onerous and narrow focused. We were doing something else that wasn’t pure ET and it wasn’t traditional scripted testing. It turns out test quest was a perfect repository for everything that we needed to be done. Also, you didn’t finish the quest until you cleaned up data, entered bugs and other things people might find unpleasant after a test session or two. There is more here on quests: Test Quests – Gamification Applied to Software Test Execution

As I point out in that post, Chore Wars is interesting, but it was challenging for sustained testing because of different personalities and motivations of different people. So we used some ideas from ARGs to sprinkle within our process rather than use it as a foundation. Certain gamer types are attracted to things like Chore Wars, but others are turned off by them, so you have to be careful with a productivity tool.

We set up a reward system that reminded people to do a more thorough job. Was there a risk assessment? Were there coverage outlines? Session sheets? How were they filled out? Were they complete? What about bug reports? Were they complete and clear?  I fought with the architects over having a leaderboard, but eventually I relented and we reached a compromise. Superstar testers can dominate a system like this, causing others to feel demoralized and not want to try anymore. We decided to overcome that by looking at chance events, which are a huge part of what makes games fun, so no one could stay and dominate the testing leaderboard, they would get knocked to the bottom randomly and would have to work their way back up. Unfortunately, we ran into regulatory issues with the leaderboard – while we forbade the practice of ranking employees based on the tool, this sort of thing can run afoul of labor laws in some countries, so we were working on alternatives but ran out of resources before we could get it completed.

Social aspects of gaming are a massive part of online games in particular, but board games are more fun with more people too. We set up a communication system similar to a company IRC system we had developed in the past. We also designed a way to ask for help and for senior testers to provide mentoring, and like MMOs, we rewarded people who worked together more than if they worked alone. Like developer tools, we set up flags for review to help get more eyes on a problem.

We also set up a voting system so testers could nominate each other for best bug, or best bug report, best bug video, and encouraged sharing bug stories and technical information with each other within the tool.

An important design aspect was interoperability with other tools, so we designed testing products to be easily exported so they could be incorporated with tools people already use. Rather than try to compete or replace, we wanted to complement what testers were already doing in many organizations, and have an alternative to the tired and outdated test case management systems. However, if you had one of those systems, we wanted to work with it, rather than against it.

Unfortunately, we ran out of resources and weren’t able to get the tool off the ground. It had the basics of Session Tester embedded in it, with improvements and a lot of game approaches mixed in with testing fundamentals.

We learned three lessons with all of this:

  1. Co-operative game play works well for productivity over a sustained period of time, while competitive game play can be initially productive, but over time it can be destructive. Competition is something that has to be developed within a co-operative structure with a lot of care. Many people shut down when others get competitive, and rewarding for things like bugs found, or bugs fixed causes people to game the system, rather than focus on value.
  2. Each team is different, and there are different personalities and player types. You have to design accordingly and make implementations customizable and flexible. If you design too narrowly, the software testing game will no longer be relevant. If design is more flexible and customizable from the beginning, the tool has a much better chance of sustained use, even if the early champions move on to other companies. I’ve had people ask me for simple approaches and get disappointed when I don’t have a pat answer on how to gamify their testing team approach without observing and working with them first. There is no simple approach that fits all.
  3. Designing a good productivity tool is very difficult, and game approaches are much more complex than you might anticipate. There were unintended consequences when using certain approaches, and we really had to take different personality and player styles into account. (There are also labour and other game-related laws to explore.) Thin layer gamification (points, badges, leaderboards) had limited value over time and only appealed to a narrow group of people.

 If some of you are looking at gamification and testing productivity, I hope you find some of these ideas useful. If you are interested in some of the approaches we used, these Gamification Inspiration Cards are a good place to start.

fuzzing, security, tools, XML

Use of Fuzzers Helps Discover XML Security Threats

I posted about fuzzing a few days ago, and I think the tools are neat, and in the hands of good testers can be powerful. They are a nice way to augment existing security testing, to test data transfers or messaging, or to simply generate test data. However, some of my readers basically said:

Big deal. It’s a neat toy for you, or some of your clients who have the time for that sort of thing, but why would I want to use one?

The Register has posted an interesting article where fuzzers were used to discover potential security holes in XML libraries. Check it out here: XML flaws threaten ‘enormous’ array of apps. If my babblings about fuzzing don’t get your attention, maybe this article will. The potential to use fuzzers (in conjunction with other security tools and techniques) to help people catch problems that bad guys will exploit is enormous.

Codemonicon, the company cited in the article, have a lot of interesting information and expertise in this area. They have a nice introduction to fuzzing on their website. This paper by Rauli Kaksonen has a lot of technical detail on fuzzing if you’d like to learn more.

SBTM, Session Based Testing, Session Tester, tools

Announcing Session Tester

UPDATE: Unfortunately, this is now a dead project. While we had amazing initial community support, the project waned after an economic downturn. We were unable to staff up a team to carry it forward. If you would like to take up the torch and start the project up again, contact me for the source code.

We’re pleased to announce a new open source, lightweight session-based testing management tool called Session Tester. Aaron West and I have been working on this project off and on for quite a while, and finally have a public beta release. We expect the beta release will be a little rough around the edges, but we’re pleased to have created something that we hope will help make session-based testing more accessible. We are following the open source model of “release early, release often” and value your feedback. Your feedback will help shape the tool as it emerges from Beta. Bug reports and usability issues are very welcome.

It’s early days yet and the tool is quite simple. User feedback will influence what features are added to the finished product(s). That’s the beauty of the open source product development process, where the evolution of the product is transparent and guided by the user community.

Session Tester was written in Java, so it should run on any operating system that has a recent version of the Java Runtime Environment. We have tested with Windows XP, Vista, Mac OS X (Intel) and Linux.

We developed this tool as an aid for testers who are using session based testing management. (Our initial interpretation of SBTM outlined in the article is more lightweight than some might be used to.) It provides an easy way to time your sessions, remind you of your testing missions, and to record your notes. It also provides an idea primer in case you get stuck, and a testing cheatsheet to review while you are in the thick of testing and would like some ideas.

Even testers who aren’t in regulated environments and don’t take detailed notes find the tool helps them add structure to their thinking when exploratory testing.

Getting Started

Download Session Tester 0.1 here. This is a beta release, so we don’t have proper installers yet. Be sure to have a recent JRE installed (at least JRE 6, update 6), or else Session Tester might not run properly.

If you are unsure about your JRE, go to the Sun site for a new version. We last used update 11, but anything recent should work: Java SE Downloads. There are also instructions on installation and usage in the user guide. Once you have a recent JRE, just unzip into a folder, double click the executable jar file, and start testing.

This is a beta release, so we expect bug reports and usability issues, as well as feature requests. Feel free to share any of those ideas on our forum. Please contribute to the community there. We will also be looking for contributors to the project itself.

Current Features

  • a timer to keep track of your session length
  • note taking that is automatically saved and formatted into XML, with extra tags for more organization
  • optional reminders
  • mission reminder so you don’t lose focus
  • session end reminder so you can start winding your session down
  • an idea primer for those times when you get stuck when trying to generate test ideas (thanks to Michael Bolton for introducing us to oblique strategies and contributing to the current list)
  • an exploratory testing cheatsheet, supplied by Elisabeth Hendrickson (check under the Help menu)

Currently, we are building the tool for individual testers to use to take notes, manage their sessions, prime ideas, and review their session files in XML format. Eventually, we will extend this to team management.

Project Goals

Our goal with this tool is to help train testers on how to do session-based testing, and make the practice easier for everyone. We also thought hard about the idea generation part of testing, and try to use the interruptions for note taking as opportunities idea generating or focusing tools, instead of just annoyances.

Thanks

Thanks to Patrick Lightbody for generously providing the space and for supporting this project, as well as to James and Jon Bach for creating SBTM, Antony Marcano for the inspiration and useful feedback, and to Jared Quinert and Mike Kelly for their usability feedback. Thanks to Michael Bolton for the inspiration for the idea primer, and for contributing test phrases to it, and thanks to Elisabeth Hendrickson for granting us permission to provide her cheatsheet as a thinking reference for testers who are using the tool.

Questions?

Sign up and post questions, concerns, comments, queries, etc. on our clearspace discussion forum.

post-agilism, tools

Pliant Software Development

Recently, I met Tim Beck, who is behind Pliant Software Development. I like his ideas, and I support what he’s doing.

When I came across this website, I liked what I saw, and when we got together in person, I really liked what I heard. Tim is thoughtful, principled, positive and forward-looking. He wants to see software development move ahead and continuously improve. Like me, he doesn’t think we have the final word on software development yet. I was also pleased to see in his work that he is also a contextualist. You won’t get any “one-true-way” marketing from him.

Here’s an excerpt from the Pliant About page:

Again, there is no right answer. Therefore, there is no point in trying to find one. What we must do is strive to adapt our processes so that we get better at developing software. As the definition at the top of this page implies, in order to be pliant, we must bend or fold our current process to fit the situation we are facing. Since the situation is constantly changing, we have to be constantly changing our process.

One comment Tim makes frequently is this:

Are we solving the real problem?

Inspired by his description of the company where he worked that coined this phrase, and hung it up on a banner, I had an idea. This question should be printed out on a large banner in software development companies, visible to all (executives included), particularly where process or tool zealotry is reaching a fever pitch. All too often we reach for a process/tool without looking at the real problem. In many cases, we then end up with our problems being compounded. Not only does the real problem remain unaddressed, but we end up with even more systems to maintain, distracting and robbing us of productive problem-solving time. We might get so distracted while working on the “solutions”, that we have even more fog preventing us from seeing the real root of the problem. Sure, it might be more fun to adopt an ambitious test automation project, or buy that shiny new tool we’ve been looking at, but if we don’t understand the real problem, we may create an even bigger one.

Check out Tim’s work. I find his posts thoughtful as well as thought-provoking. In an industry that seems prone to silver-bullet-itis, we need more thoughtful process skeptics like Tim, who genuinely want to see the world become a better place.

tools, Watir

Quick Update

I’ve been blogging less lately; have had a lot on the go. I’m also spending more time writing code and will spare you the pain of having to read source code in a blog post.

Some quick points:

test automation, tools, Watir, web applications

Watir Release

The Web Testing with Ruby group released version 1.0 of the Watir tool today. Check it out here: Get Watir

Download the latest release which is a zip file at the top of the watir package list. (At the time of this posting, the latest version is 1.0.2.) Open the User Guide for installation instructions, and check out the examples to see how you can use the tool.

You will need Ruby if you don’t have it installed. Version 1.8.2-14 Final is a good candidate to use, or 1.8.1-12 if you prefer to be more on the trailing edge.

For those who may not be familiar with the tool, it allows for testing web applications at the web browser level. The project utilizes testable interfaces in web browsers to facilitate test automation. Currently it only supports Internet Explorer, but plans are underway to support other browsers. It is also only available on Windows.

ruby, test automation, tools

Green Bar in a Ruby IDE!

The Ruby Development Tool project team, who develop a plugin for Eclipse, have added some great features in their latest release. Most important (to me), is test::unit support, as well as some other features such as code-completion based on configurable templates, and a regular expression plugin.

I’ve been using RDT in Eclipse for a few months, and with this latest release, I’m very pleased to finally have a green bar in my Ruby development. Or more often in my case, a red bar.

Thanks to Glenn Parker for letting me know about this release.

test cases, tools

Dehumanizing Software Testing

I was talking with Daniel Gackle, a software developer and philosophical thinker, about developing software and systems. Daniel mentioned in passing that we often forget about the humanity in the system – the motivation of the human who is using the tool, or doing the work. This comment resonated with me, particularly with regards to some software testing practices.

Sometimes when we design test plans, test cases, and use testing tools, we try to minimize the humanity of the tester. The feeling might be that humans are prone to error, and we want solid, repeatable tests, so we do things to try to minimize human error. One practice is the use of procedural test scripts. The motivation often is that if the test scripts are detailed enough, anyone can repeat them, with a benefit that there will be less chance for variability or error. The people who write these tests try to be as detailed as possible with one motivation being that the test must be repeated the same way each time. It is not desirable to have variation because we want this exact test case to be run the same way every time it is run.

This type of thinking spills over into test automation as well. There is a push to have tools that can be used by anyone. These tools take care of the details, all a user needs to do is learn the basics and point and click and the tool does the rest for us. We don’t need specialists then, the tool will handle test case design, development and execution.

Without going into the drawbacks that both of these approaches entail, I want to focus on what I would call the “dehumanizing of software testing”. When we place less value on the human doing the work, and try to minimize their direct interaction with software under test, what do we gain, and what do we lose?

In The Dumbing Down of Programming, Ellen Ullman describes some of what is lost when using tools that do more work for us. Ullman says:

the desire to encapsulate complexity behind a simplified set of visual representations, the desire to make me resist opening that capsule — is now in the tools I use to write programs for the system.

It is today, (as it was in 1998 when this article was written), also in procedural test cases (or scripts) and in the popular “capture/replay” or event recording testing tools. Both seek to “encapsulate complexity” behind an interface. Both practices I would argue, lead to dehumanizing software testing.

Ullman mentions that the original knowledge is something we give up when we encapsulate complexity:

Over time, the only representation of the original knowledge becomes the code itself, which by now is something we can run but not exactly understand.

When we don’t think about the human interaction that I believe is so valuable to software testing, we need to be aware of what we are giving up. There is a trade-off between having interactive, cognitive testing, and testing through interfaces where the complexity is hidden from us. There are cases where this is useful, we don’t always need to understand the underlying complexity of an application when we are testing against an interface. However, as Ullman says, we should be aware of what is going on:

Yet, when we allow complexity to be hidden and handled for us, we should at least notice what we’re giving up. We risk becoming users of components, handlers of black boxes that don’t open or don’t seem worth opening.

When is hiding the complexity dehumanizing software testing, and when is it a valuable practice? I would argue that as soon as we encapsulate complexity to the point that the tester is discouraged from interacting with the software with their mind fully engaged, we are dehumanizing software testing. If a tester is merely following a script for most of their testing work, they are going to have trouble being completely mentally engaged. If a testing tool is trying to do the work of test design and execution, it will be difficult for the tester to be completely engaged with testing. I believe we miss out on their skills of observation, inquiry and discovery when we script what the tester needs to do and they do most of their work by rote.

The important thing to me is to understand what the trade-offs are before we adopt a testing practice. When we forget the humanity of the system, we will probably get results other than what we intended. We need to be aware of how people impact the use of tools, and how people work on projects, because after all, it’s still the people who are key to project success.

agile, agile testing, ruby, test automation, tools, Watir

Automating Tests at the Browser Layer

With the rise in popularity of agile development, there is much work being done with various kinds of testing in the software development process. Developers as well as testers are looking at creative solutions for test automation. With the popularity of Open Source xUnit test framework tools such as JUnit, NUnit, HTTPUnit, JWebUnit and others, testing when developing software can be fun. Getting a “green bar” (all automated unit tests in the harness passed) has become a game, and folks are getting more creative about bringing these test tools to new areas of applications.

One area of web applications that is difficult to test is the browser layer. We can easily use JUnit or NUnit at the code level, we can create a testable interface and some fixtures for FIT or FITnesse to drive the code with tabular data, and we can run tests at the HTTP layer using HTTPUnit or JWebUnit. Where do we go from there, particularly in a web application that relies on JavaScript and CSS?

Historically, the well-known “capture/replay” testing tools have owned this market. These are an option, but do have some drawbacks. Following the home brew test automation vein that many agile development projects use, there is another option using a scripting language and Internet Explorer.

IE can be controlled using its COM interface (also referred to as OLE or ActiveX) which allows a user to access the IE DOM. This means that all users of IE have an API that is tested, published, and quite stable. In short, the vendor supplies us with a testable interface. We can use a testable interface that is published with the product, and maintained by the vendor. This provides a more stable interface than building one at run-time against the objects in the GUI, and we can use any kind of language we want to drive the API. I’m part of a group that prefers the scripting language Ruby.

A Simple Example

How does it work? We can find methods for the IE COM On the MSDN web site, and use these to create tests. I’ll provide a simple example using Ruby. If you have Ruby installed on your machine, open up the command interpreter, the Interactive Ruby Shell. At the prompt, enter the following (after Brian Marick’s example in Bypassing the GUI, what we type is in bold, while the response from the interpreter is in regular font).

irb> require ‘win32ole’
=>true
irb> ie = WIN32OLE.new(‘InternetExplorer.Application’)
=>#<WIN32OLE:0x2b9caa8>
irb> ie.visible = true
#You should see a new Internet Explorer application appear. Now let’s direct our browser to Google:
irb> ie.navigate(“http://www.google.com”)
=>nil
#now that we are on the Google main page, let’s try a search:
irb> ie.document.all[“q”].value = “pickaxe”
=>”pickaxe”
irb> ie.document.all[“btnG”].click
=>nil

#You should now see a search returned, with “Programming Ruby” high up on the results page. If you click that link, you will be taken to the site with the excellent “Programming Ruby” book known as the “pickaxe” book.

Where do we go from here?

Driving tests this way through the Interactive Ruby Shell may look a little cryptic, and the tests aren’t in a test framework. However, it shows us we can develop tests using those methods, and is a useful tool for computer-assisted exploratory testing, or for trying out new script ideas.

This approach for testing web applications was pioneered by Chris Morris, and taught by Bret Pettichord. Building from both of those sources, Paul Rogers has developed a sophisticated library of Ruby methods for web application testing. An Open Source development group has grown up around this method of testing first known as “WTR” (Web Testing with Ruby). Bret Pettichord and Paul Rogers are spearheading the latest effort known as WATIR. Check out the WATIR details here, and the RubyForge project here. If this project interests you, join the mailing list and ask about contributing.

*This post was made with thanks to Paul Rogers for his review and corrections.

ruby, tools, Watir, web applications

Visible Tests and Web Testing with Ruby

Announcing a new project and blogging my notes for the Scripting Web Tests Tutorial which I prepared for XP Agile Universe:

Visible Tests

Automated test results are often difficult to communicate to business stakeholders on a project. People frequently ask me about the visibility of tests on a project saying: “The customer doesn’t understand JUnit tests” or “The JUnit and FIT tests mean very little to the customer. The customer doesn’t usually think in terms of green bars, or table-driven tests like developers and technical testers do. They believe we are testing, but they can’t see the tests run in a way that they relate to. How do we raise the visibility of tests?”

The WTR IE Controller has an advantage in test visibility since the testable interface we use is about as close as we can get to interacting with a program the way an end-user would. Business stakeholders understand testing at the GUI layer because that’s the way they relate to the program. These tests can be played back to the business stakeholders so they can see the tests run in a way they relate to. Technical people on a project team relate to the program at various layers and sometimes we forget about the business problems we are solving. Techies look at the backend and the front-end, business users usually only see and understand the front-end of the application. The preferred method of interaction with the program can differ between the groups.

Business stakeholders can watch the tests play back on a computer which provides rich, visual feedback. If the test fails, they see it fail. If it passes, they see it pass while exercising the application in a way that they would themselves. While they have faith in the ability of the technical members of the team, and will accept the testing numbers and their word, nothing replaces the assurance they get from seeing it work, and from manipulating the product themselves. With IE Controller tests, the customer can see if the tests pass or fail by watching how the application works in the Internet Explorer browser. Tests can also be designed to mimic business interaction, and provide results logging that non-technical project stakeholders can understand. The tests will demo the business solutions that the customer needs the application for in the first place. These kinds of automated tests help provide more assurance in what the technical members of the team are doing.

At an end of iteration meeting, the application can be demonstrated using these tests if the group wanted to give a quick demonstration.

Announcing WATIR

The latest version of the Web Testing with Ruby project has begun under the WATIR project spearheaded by Paul Rogers and Bret Pettichord. I’m excited by the prospect of a more sophisticated Open Source web testing solution using Ruby. I’ve had success with the IE Controller tests I’ve written to date, and look forward to using a more complete solution.