Collaborative Software Testing

Give Testers Power! An Interview with Test Common

Test Common editor Bruce Daley commented on my "Man and Machine" article and then interviewed me about it last week. The interview is here: An Interview with Jonathan Kohl. We talked about automated testing issues, and giving testers power with automation tools, instead of making them feel like they are being replaced.

Software Development Process Fusion - Part 1

I first brought this idea up publicly last year at the Agile Vancouver conference with a promise that I would share more of my thoughts. What follows is an attempt to fulfill that promise. This has turned out to be rather long, so it will appear as a blog series.

I grew up in an environment with a lot of music. My grandfather had a rare mastery over a wide variety of musical instruments, and family gatherings were full of singing and impromptu jams. At home, my father had a very eclectic taste in music, and I had a steady diet of gospel, classical, big band, bluegrass, traditional German and Celtic music. One of my babysitters had spent most of her life in India, and introduced me to all kinds of wonderful forms of Indian classical music when I was very small. I was exposed to popular music on the radio, and I took part in various music groups in school bands, choirs and at church. By the time I was in high school, I had a wide exposure to many different kinds of music, and enjoyed any sort of music that moved me, no matter what the style. I could enjoy a common thread in music that was composed and performed in a way that appealed to me, even if the styles were very different. In some cases, my classical friends couldn't stand some of the popular music I enjoyed, and some of my gospel music fans would refuse to listen to secular music. Enjoying a wide variety of music styles could be controversial, depending on who I was talking to.

In the late 80s, I was in a choir that was in a competition in Toronto. I was billeted with a family who introduced me to a Canadian band called "Manteca." My new friends introduced me to a style of music called "fusion," and Manteca were well-known for their mastery of that style. There were elements of improvisational jazz, popular music and world music in their work. Because of Manteca, I decided to learn more about the history of this style. It didn't take long before I discovered Miles Davis recordings from the '60s and '70s that pioneered a combination of musical styles. I then checked out work by Larry Coryell, and jazzy popular bands like Chicago and Blood Sweat & Tears. From Miles Davis, I followed some of his former band members works such as John McLaughlin, Herbie Hancock, Chick Corea and Joe Zawinul. I also checked out groups like the Crusaders, Weather Report and anything with bassist Jaco Pastorius. Some of the music was highly experimental, sometimes it was hard to listen to. One of my favorite bands in that style was Mahavishnu Orchestra, founded by wizard guitarist John McLaughlin. McLaughlin also founded a band called Shakti that utilized a different style of fusion. Shakti was a highly improvisational band utilizing master musicians from India, and McLaughlin on guitar.

From this musical journey, I discovered progressive music from the '70s, with bands like Yes, King Crimson, Emerson Lake and Palmer, and Genesis. These were all groups with highly talented musicians who brought other musical styles into popular music forms. As with fusion, this style of music was highly experimental - some of it made popular charts, and others remained obscure. What these styles of music share is a demanding level of skill for the performers, an experimental, pushing the envelope attitude, often utilizing collective improvisation (particularly in live concerts.) They were also controversial when they first came out, but many "wild in their time" elements have become enmeshed in mainstream music today. However, in the pioneering days of fusion, it was not uncommon for critics to pan albums and for purists to cry fowl.

One musician who has had an enormous influence on popular music is former Genesis front-man Peter Gabriel. By the late 1990s, the style of music that Peter Gabriel made accessible to a huge audience in the 1980s emanated from airwaves and stereos everywhere you went. I heard Jesse Cook talk (a guitarist who fuses flamenco guitar with many different styles) about the impact Peter Gabriel has had on modern music, particularly with his ability to fuse popular music with traditional music from other parts of the world ("world music".) Jesse Cook mused at the time that anything we heard on the radio could probably be traced back to Gabriel. When we were discussing the different styles that had an impact on everyday popular music, we wondered what it was like for the musical pioneers when their ideas were new, and how little most of us know about the history of music we take for granted.

I still listen to music that combines different styles. One of my recent discoveries is Harry Manx, a blues guitarist who plays slide guitar on a Mohan Vina, an Indian slide guitar. He deftly fuses traditional Indian music with traditional blues. I find that I am moved by too many styles of music to merely just choose one, and sometimes the weird combinations do something for me that just one style on its own can't do. I still love classical music, and find that a particular period or style of music suits my mood. Music can touch us in ways other things can't. Music also evolves, and musicians draw on many influences - yesterday's "pure" style becomes influenced by something else, and we co-opt other ideas and change. I suppose we expect that in the arts. For example, the Canadian artist Emily Carr's work is called "post-impressionist" because she came after the famous impressionist painters, and developed a unique style that doesn't quite fit in that category. Like the fusion musicians, Carr's artistry has many influences and changed a good deal over her life. Carr had a special ability to fuse disparate themes together in a painting. She might combine everyday objects we might see in our homes with a nature scene, or combine two different scenes together.

In software development, we don't have a long and rich history to draw from like our artistic counterparts. That doesn't stop me from approaching software development from the same angle as I do anything else though. Brian Marick in his "Process and Personality" article says: "...my methodological prescriptions and approach match my personality." This is also true with me. I like to fuse different ideas together and see if I can create something new from the combination. There may be stark lines drawn between the fields where the ideas come from, but that doesn't bother me too much. It gets me into trouble sometimes, but the ideas are what are important to me. When it comes to software development, I don't really care if an idea is "Agile", "waterfall" or has no label at all. If it's a good idea to me, it's a good idea. Sometimes on software development projects, I weave together combinations of these ideas in a way that may seem strange to some. I've started calling this style that I and others are exploring: "software development process fusion."

Exploratory Testing: More than Superficial Bug Hunting

Sometimes people define exploratory testing (ET) quite narrowly, such as only going on a short-term bug hunt in a finished application. I don't define ET that narrowly, in fact, I do ET during development whether I have a user interface to use or not. I often ask for and get some sort of testing interface that I can use to design and execute tests around before a UI appears. I'll also execute traditional black-box testing through a UI as a product is being developed, and at the end, when development feels it is "code complete". I'm not alone. Cem Kaner mentioned this on the context-driven testing mailing list, which prompted this blog post.

Cem wrote:

"To people like James and Jon Bach and me and Scott Barber and Mike Kelly and Jon Kohl, I think the idea is that if you want useful exploratory testing that goes beyond the superficial bugs and the ones that show up in the routine quicktests (James Whittaker's attacks are examples of quicktests), then you want the tester to spend time finding out more about the product than its surface and thinking about how to fruitfully set up complex tests. The most effective exploratory testing that I have ever seen was done by David Farmer at WordStar. He spent up to a week thinking about, researching, and creating a single test-which then found a single showstopper bug. On this project, David developed exploratory scenario tests for a database application for several months, finding critical problems that no one else on the project had a clue how to find. "

In many cases, when I am working on a software development project, a good deal of analysis and planning go into my exploratory testing efforts. The strategies I outline for exploratory testing reflect this. Not only can they be used as thinking strategies in the moment, at the keyboard, testing software, but they can guide my preparation work prior to exploratory testing sessions. Sometimes, I put in a considerable amount of thought and effort modeling the system, identifying potential risk areas and designing tests that yield useful results.

In one case, a strange production bug occurred in an enterprise data aggregation system every few weeks. It would last for several days, and then disappear. I spent several days researching the problem, and learned that the testing team had only load tested the application through the GUI, and the real levels of load occurred through the various aggregation points communicating in. I had a hunch that there were several factors at work here and it took time to analyze them. It took several more days working with a customer support representative who had worked on the system for years before I had enough information to work with the rest of the team on test design. We needed to simulate not only the load on the system, but the amount of data that might be processed and stored over a period of weeks. I spent time with the lead developer, and the lead system administrator to create a home-grown load generation simulation tool we could run indefinitely to simulate production events and the related network infrastructure.

While the lead developer was programming the custom tool and the system administrator was finding old equipment to set up a testing environment, I created test scenarios against the well-defined, public Web Services API, and used a web browser library that I could run in a loop to help generate more light load.

Once we had completed all of these tasks, started the new simulation system, and waited for had the data and traffic statistics to be at the level that I wanted to generate, I began testing. After executing our first exploratory test, the system fell over, and it took several days for the programmers to create a fix. During this time, I did more analysis and we tweaked our simulation environment. I repeated this with the help of my team for several weeks, and we found close to a dozen show-stopping bugs. When we were finished, we had an enhanced, reusable simulation environment we could use for all sorts of exploratory testing. We also figured out how to generate the required load in hours rather than days with our home-grown tools.

I also did this kind of thing with an embedded device that was under development. I asked the lead programmer to add a testable interface into the new device he was creating firmware for, so he added a telnet library for me. I used a Java library to connect to the device using telnet, copied all the machine commands out of the API spec, and wrapped them in JUnit tests in a loop. I then created code to allow for testing interactively, against the API. The first time I ran a test with a string of commands in succession in the IDE, the device failed because it was writing to the input, and reading from the output. This caused the programmer to scratch his head, chuckle, and say: "so that's how to repeat that behavior..."

It took several design sessions with the programmer, and a couple days of my time to be able to set up an environment to do exploratory testing against a non-GUI interface using Eclipse, a custom Java class, and JUnit. Once that was completed, the other testers used it interactively within Eclipse as well. We also used a simulator that a test toolsmith had created for us to great effect, and were able to do tests we just couldn't do manually.

We also spent about a week creating test data that we piped in from real-live scenarios (which was a lot of effort to create as well, but well worth it.) We learned a good deal from the test data creation about the domain the device would work in.

Recently, I had a similar experience - I was working with a programmer who was porting a system to a Java Enterprise Edition stack and adding a messaging service (JMS.) I had been advocating testability (visibility and control - thanks James) in the design meetings I had with the programmer. As a result, he decided to use a topic reader on JMS instead of a queue so that we can see what is going on more easily, and added support for the testers to be able to see what the Object-Relational Mapping tool (JPA) is automatically generating map and SQL-wise at run-time. (By default, all you see is the connection information and annotations in Java code, which doesn't help much when there is a problem.)

He also created a special testing interface for us, and provided me with a simple URL that passes arguments to begin exercising it. For my first test, I used JMeter to send messages to it asynchronously, and the system crashed. This API was so far below the UI, it would be difficult to do much more than scratch the surface of the system if you only tested through the GUI. With this testable interface, I could use several testing tools as simulators to help drive my and other tester's ET sessions. Without the preparation through design sessions, we'd be trying to test this through the UI, and wouldn't have near the power or flexibility in our testing.

Some people complain that exploratory testing only seems to focus on the user interface. That isn't the case. In some of my roles, early in my career, I was designated the "back end" tester because I had basic programming and design skills. The less technical testers who had more knowledge of the business tested through the UI. I had to get creative to ask for and use testable interfaces for ET. I found a place in the middle that facilitated integration tests , while simulating a production environment, which was much faster than trying to do all the testing through the UI.

I often end up working with programmers to get some combination of tools to simulate the kinds of conditions I'm thinking about for exploratory testing sessions, with the added benefit of hitting some sort of component in isolation. These testing APIs allow me to do integration tests in a production-like environment, which complements the unit testing the programmers are doing, and the GUI-level testing the black box testers are doing. In most cases, the programmers also adopt the tools and use them to stress their components in isolation, or as I often like to use them for, to quickly generate test data through a non-user interface while still exercising the path the data will follow in production. This is a great way to smoke test minor database changes, or database driver or other related tool upgrades. Testing something like this through the UI alone can take forever, and many of the problems that are obvious at the API level are seemingly intermittent through the UI.

Exploratory testing is not limited to quick, superficial bug hunts. The learning, analyzing, executing, test idea generation and execution are parallel activities, but sometimes we need to focus harder in the learning and analyzing areas. I frequently spend time with programmers helping them design testable interfaces to help with exploratory testing at a layer behind the GUI. This takes preparation work including analysis and design, and testing of the interface itself, which all help feed into my learning about the system and into the test ideas I may generate. I don't do all of my test idea generation on the fly, in front of the keyboard.

In other cases, I have tested software that was developed for very specialized use. In one case, the software was developed by scientists to be used by scientists. It took months to learn how to do the most basic things the software supported. I found some bugs in that period of learning, but I was able to find much more important bugs after I had a basic grasp of the fundamentals of the domain the software operated in. Jared Quinert has also had this kind of experience: "I've had systems where it took 6 months of learning before I could do 'real' testing."