Collaborative Software Testing

Modeling Web Applications

When I am testing web applications, I like to analyze the system to find areas of testability, and to identify potential areas of instability. To help remember areas to investigate, I use this mnemonic:
FP DICTUMM

F - Framework (Struts, Rails, Plone, Spring, .NET, etc.)
P - Persistence (Hibernate, Toplink, iBatis, Active Record, DAOs, etc.)
D - Data (database, flat files, etc.)
I - Interfaces (code level, web services, user, etc.)
C - Communication (HTTP(S), web services, SOAP, XMLHTTPRequest, database drivers, etc.)
T - Technology (.NET, J2EE, Ruby, PHP, Perl etc.)
U - Users (Human end users, and other systems or apps that use this system)
M - Messaging (JMS, Enterprise Message Beans, Web Methods, etc.)
M - Markup (HTML, CSS, DHTML, XML, XHTML)

Each of these areas help inform my testing, or provide areas of testability. Understanding the whole picture from the data model up to the HTML displayed in a web browser is important for my testing.

Who Do We Serve? - Intermittent Problems Revisited

I seeded a problem for readers to spot in my recent post "Who Do We Serve?". George Dinwiddie was the only reader who not only spotted the problem, he emailed me. George says:

"...there's [a] red flag that occurs early in your story that I think deserves more attention. "Sometimes the tests fail sporadically for no apparent reason, but over time the failures appear less frequently." As Bob Pease (an analog EE) used to say in Electronic Design and EDN magazines, "If you notice anything funny, record the amount of funny." When something fails intermittently or for an unknown reason, it's time to sit up and take notice. The system should not be doing something the designers don't understand."

George spotted the problem that I introduced, but didn't expand on, and as I hoped a reader would do, emailed me to point out my omission. Good job George, and thanks for emailing me when you spotted it. Your thoughts are absolutely in line with my experience.

One area of expertise I have developed is the ability to track down repeatable cases for intermittent bugs. These are difficult problems for any team to deal with. They don't fit the regular model, often requiring off-the-wall experimentation, lots of thinking time, tons of creativity, and a good deal of patience. One needs to look at the potentially enormous amount of data in front of them, and figure out how to weed out the things that aren't important. There is trial and error involved as well, and a good deal of exploration. Naturally, exploratory testing is a fit when tracking intermittent problems.

On Agile teams, I have found we can be especially prone to ignoring intermittent problems. The first time I discovered an intermittent problem on a team using XP and Scrum, we did exactly as Ron Jeffries has often recommended: we came to a complete stop, and the whole team jumped in to work on the problem. We worked on it for a day straight, but people got tired. We couldn't track down a repeatable case, and we couldn't figure out the problem from the code. After the second day of investigation, the team decided to move on. The automated tests were all passing, and running through the manual acceptance tests didn't reveal the problem. In fact, the only person who regularly saw the problem was me, the pesky tester, who shouldn't have been on an XP team to begin with.

I objected to letting it go, but the team had several reasons, and they had more experience on Agile projects than I did. Their chief concerns were that tracking down this problem was dragging velocity down. They wanted to get working on the new story cards so we could meet our deadline, or finish early and impress the customer. Their other concern was that we weren't following the daily XP processes when we were investigating. They said it would be better to get back into the process. Furthermore, the automated tests were passing, and the customer had no problem with the functionality. I stood down, and let it go, but made sure to watch for the problem with extra vigilance, and took it upon myself to find a repeatable case.

Our ScrumMaster talked to me and said I had a valid concern, but the code base was changing so quickly that we had probably fixed the bug already. This was a big red flag to me, but I submitted to the wishes of the team and helped test newly developed stories and tried to give the programmers feedback on their work as quickly as I could. That involved a lot of exploratory testing, and within days, my exploratory testing revealed that the intermittent bug had not gone away. We were at the end of a sprint, and it was almost time for a demo for all the business stakeholders. The team asked me to test the demo, run the acceptance tests, and support them in the demo. We would look at the intermittent bug as soon as I had a repeatable case for it.

Our demo was going well. The business stakeholders were blown away by our progress. They thought it was some sort of trick - it was hard for them to understand that in so short a time we had developed working software that was running in a production-like environment. Then the demo hit a snag. Guess what happened? The intermittent bug appeared during the demo, on the huge projector screen in front of managers, executives, decision makers and end users. It was still early in the project, so no one got upset, but one executive stood up and said: "I knew you guys didn't have it all together yet." He chuckled and walked out. The sponsor for the project told us that the problem better be fixed as soon as possible.

I don't think management was upset about the error, but our pride was hurt. We also knew we had a problem. They were expecting us to fix it, but we didn't know how. Suddenly the developers went from saying: "That works on my machine." to: "Jonathan, how can I help you track down the cause?" In the end, I used a GUI automated testing tool as a simulator, and based on behavior I had seen in the past, and patterns I documented from the demo, I came up with a theory, and using exploratory testing, I designed and executed an experiment. Using a GUI test automation tool as a simulator while running manual test scenarios helped me find a cause quite quickly.

My experiment failed in that it showed my theory was wrong, but a happy side effect was that I found the pattern that was causing the bug. It turned out that it was an intermittent bug in the web application framework we were using. The programmers wrote tests for that scenario, wrote code to bypass the bug in the framework, and logged a bug with the framework project. At the next demo, we ran manual tests and automated tests showing what the problem was, and how we had fixed it. The client was pleased, and the team learned a valuable lesson. We learned that an XP team is not immune to general software problems, and as George said we practiced this in the future: "When something fails intermittently or for an unknown reason, it's time to sit up and take notice. "

Other Agile teams I've been on have struggled with intermittent errors, particularly teams that had high velocity and productivity. Regular xUnit tools and acceptance tests aren't necessarily that helpful, unless they are used as part of a larger test experiment. Running them until they turn green, and then checking code in without understanding the problem and fixing it will not cause it to go away. An unfixed intermittent bug sits in the shadows, ticking away like a time bomb, waiting for a customer to find it.

Intermittent problems are difficult to deal with, and there isn't a formula to follow that guarantees success. I've written about them before: Repeating the Unrepeatable Bug, and James Bach has the most thorough set of ideas on dealing with intermittent bugs here: How to Investigate Intermittent Problems. I haven't seen intermittent bugs just disappear on their own, so as George says, take the time to look into the issue. The system should not be doing something the designers don't understand.

Responses to The Agile Backlash

Here are some reactions to the Agile Backlash post:

Tim Beck weighs in:

"In the end, I firmly believe (and I think Jonathan does too) that the "Agile or not" debate is moot. There is a post-agilism movement coming (or already here) and we are going to have to deal with it. The software world is changing (did it ever stop) and if you don't adapt to the world you'll get left behind."

As Tim knows, "post-agilism" is a descriptive term of a growing community I have observed over the past couple of years. I am not trying to found a movement, there is no manifesto, and I'm not selling a process. "Post-agilism" is a term I used to describe the growing group of practitioners who agree with the Agile Manifesto, but don't consider themselves to be "Agile" anymore. I struggled with this for a while, and I find the term helpful for my own thinking. It was difficult to square the Agile movement on one hand, and this group of practitioners I kept meeting who just didn't fit. Others have described a watershed moment when they read the term. "That describes me! I'm not alone!"

Jason Gorman is on the money:

"I'm saying that I completely believe in iterative and incremental, feedback-driven delivery of working software. This is based not only on experience, but also on some simple maths that shows me that the evolutionary approach is nearly always better and never worse than single-step delivery ("waterfall").

But I'm not saying that because it's written in a book or on a web site. And that's what makes me post-Agile, I guess."

A reader emailed me:

"This is the most balanced Agile piece I remember you writing, the most balanced IT piece overall. The focus has moved from just Agile to software development, which is where I wish the industry focus would move."

Another reader emails:

"All the religious hype really gets up my nose, and I just want to get on with the business of developing software the best way we know how, and thinking about all the ways we can do it better."

Another reader says:

"I'm not yet ready to abandon the term "Agile" to those who invoke it in name only. I'd rather shine the light of day on this practice. But whatever name you use, keep up the good fight."

This is exactly the kind of thing I like to see from Agilists."I'm not ready to abandon the term. If you are, fine, but I'm going to work on improving things from within the community than from without."

Chris McMahon expands on the skill component:

"Skill of course is critically important. (The Economist had an article a couple of weeks ago pointing out that even while companies are outsourcing function to India and China, they are desperate for talent, and will hire talented employees anywhere in the world they find them.)
Other *human* attributes are also critically important."

Matt Heusser posts:

"I see a lot of "Agile" process zealots. I don't get it. One of the points of the manifesto is to focus on individuals and interactions over process and tools, and yet people keep selling methodologies and tools. I suppose that's because it's easy to sell. I intend to propose a session at the Agile Conference 2007 to discuss this."

Oddly enough, the term "post-agilism" came to my mind after talking with Matt over a year ago, after we both did keynotes at a conference. Matt was wondering whether the Agile movement was an example of a type of post-modernism. I thought about it for a couple of months, but there was this growing group of people I met that didn't fit that seemed to be more like post-modernists than the Agilists I was observing.

I also felt conflicted. I liked the Agile Manifesto, and the values of XP, but I didn't consider myself an Agilist anymore. I had been on teams that practiced evolutionary design and development before the "Agile" term was formed, and I just didn't relate to the Agile movement anymore. I felt it had stopped innovating. I wasn't alone. Across the pond, at approximately the same time I was thinking of these things, Jason Gorman was thinking the exact same thing. Jason expressed the post-modernism angle much better than I could. (Note: please work to prove us wrong.)

Matt has jumped ahead with this idea of Post Modern Software Development , and is talking about Perils and Pitfalls of Agile Development. Keep pushing the envelope Matt!

As Tim Beck says, there is an Agile backlash which is already taking place in some communities, or is on its way. Our choice is to either deal with it in a healthy way, or get swept aside when the charlatans jump on the next fad tool or process they can sell to the unwitting.

Another reader commented:

"Agile is yesterday's silver bullet."

This is an interesting point, with potentially serious implications. If "Agile" is perceived by some to be a silver bullet, we are going to have to fight to keep the good things the Agile movement popularized from getting thrown out with the bad.

One reader said to me: "This is great! There is nothing better to improve the Agile movement or help create something new than to speak from the heart and buck the groupthink and hype with thoughtful introspection and constructive criticism. Furthermore, dealing with a backlash in a healthy way is extremely important." This reader then told me a story of a company he had dealt with that used an Agile consultancy for a project. The project failed, so management banned anything that could be associated with "Agile". What that meant was that there was a ban imposed on anything that used an iterative life cycle, and incremental, evolutionary design and delivery. This is exactly the kind of reaction we don't want to see. Instead of building on what worked and improving, they have taken software development back into the dark ages in that company. My reader went on: "This isn't the only company I've seen do this - throwing out "Agile" and returning to big up-front design. I wish more people understood that there is a better way, and I think what you are doing sheds light on that. This can help us improve." My hope with the Agile Backlash post is to let people know there is a better way.