Category Archives: web applications

Modeling Web Applications

When I am testing web applications, I like to analyze the system to find areas of testability, and to identify potential areas of instability. To help remember areas to investigate, I use this mnemonic:

FP DICTUMM

F – Framework (Struts, Rails, Plone, Spring, .NET, etc.)
P – Persistence (Hibernate, Toplink, iBatis, Active Record, DAOs, etc.)

D – Data (database, flat files, etc.)
I – Interfaces (code level, web services, user, etc.)
C – Communication (HTTP(S), web services, SOAP, XMLHTTPRequest, database drivers, etc.)
T – Technology (.NET, J2EE, Ruby, PHP, Perl etc.)
U – Users (Human end users, and other systems or apps that use this system)
M – Messaging (JMS, Enterprise Message Beans, Web Methods, etc.)
M – Markup (HTML, CSS, DHTML, XML, XHTML)

Each of these areas help inform my testing, or provide areas of testability. Understanding the whole picture from the data model up to the HTML displayed in a web browser is important for my testing.

Combating Model Myopia in Testing

Part 1

In my last post, I posed a testing problem. This post begins to address how I view the problem, and my motivation behind posing it.

Models

In software testing, a “model” is a representation of something we are trying to understand. It might be a representation in our mind (an idea), a diagram (like an Entity-Relationship diagram), or it might be a physical representation. A model helps us think of a system or concept in a certain way, and as a representation is not going to be perfect or complete. We can have many models for the same thing – our only limitation is our imagination.

Here’s an example:

How many ways can you model a web page? I can think of at least three categories:

  1. as an end-user, or consumer view where I am using a web page for some purpose. I see the webpage as having text, colors, pictures and hyperlinks. I visualize that web page, served up in a web browser when I think about a particular page I find useful, and want to go to it. I interface with it and relate to it as it is shown through a web browser.
  2. as a web developer. I tend to develop web pages using a text editor and do all of the HTML, CSS, etc. by hand. I interface with the web page with a text editor, and I visualize it and relate to it by the HTML view which is made up of markup tags. I interface with the web page primarily through a text editor.
  3. as a web designer. I either have a visual picture in my mind of the layout and design of a web site I want to create and I follow that, or sometimes I draw it out using a tool. Even a rough sketch of a general layout on a piece of paper or on a whiteboard will convey that idea. I visualize the web page according to that layout.

These are three simple examples of modeling a web page. I added a couple of dimensions to help describe the models – a user, and an interface. Each one is different, but I may switch between all three as I’m developing a web page, or when I’m testing.

Since I often test web applications, I think of these different views a lot. I think of the first two very often, especially since I did a lot of web development in the mid-late ’90s. When I started writing web applications, I also learned to model from a programming context. My HTML tags, page layout, and content would be generated according to program logic, so I had a fourth model category derived from my role as a web application programmer. Understanding all the components used in rendering a web page is important to me when testing a web application. Everything from the web browser to the application server, web server, database if one is used, any drivers that are used, as well as the program code design is important information that can help with testing. I actually visualize what is happening across the entire application path when data is input into a web page, sent to the web server, processed, and the page I am on is updated. Even a simple HTML page has an interesting path. From the web server we have a text document that contains HTML markup, with certain tags that denote it as such. When a web browser asks for that page, the web server takes the tags, processes them and sends them via HTTP to the machine that is asking for it. The web browser takes that HTTP response, interprets it, and if it comes through without an error, it processes the HTML and displays it in a web browser.

Why is this important?

Modeling applications like this helps me imagine areas of potential problems based on experience and technical knowledge of the various technologies used, and also helps me narrow down or get repeatable cases for bugs I find in the application. It also helps me generate different testing ideas instead of one “happy path” based off of requirements documents, I can think of other potential usage of the software that might have the potential for problems. When I work with software users in their own environment, I am always amazed at how they use the software in unintended ways which they are perfectly happy doing.

What other models can you think of when testing web applications? Who are the users? Not all of them will be human. What are the interfaces? Not all of them will involve a Graphical User Interface like most modern, familiar web browsers.

But I’ve Run Out of Ideas!

The problem I described was something I deliberately chose as a modeling problem. Notice that in the energy industry, when I got a failure on a search attempt that contained the word “well”, I immediately had a suspicion of what the problem might be. But if I was in another industry, “well” may not be in my common vocabulary as a noun, but primarily an adverb. (Note that language is important when testing.) In my description here, we have at least two modeling categories of the software. One as “enterprise search”, and one as “my energy company search”. The ideas that each model represent are going to be subtly different. That subtle shift between models can be the difference between tracking down the source of a bug or not. But if I didn’t have energy experience, how can I find that model that will help me track down the bug?

I call this so-called running out of ideas “testing model myopia”. If we were omniscient, we would know all the possible ways to model the software, and we’d know where all the bugs were. If we were semi-omniscient and merely knew all the possible ways to model our software, we’d have an easier time. Since we aren’t, we have to use our skill as testing thinkers to break out of mental blocks. This can be very difficult if the entire team has testing model myopia as I described in the problem. When under project stress, with constraints on a development team such as time and resources, a tester may have to build a case in order to be able to pursue a testing idea. The predominant model of the software as shared by the team can be very entrenched, and it can be threatening to have that model questioned or analyzed. Nevertheless, as a tester, we are what Jerry Weinberg has described: “someone who can imagine that there might be another way.”

One of the first places I would start to tackle this problem is to imagine other models of the software that we aren’t addressing when testing.

Stay tuned for Part 2.

Testing an Application in Layers

There is often debate about test automation versus manual testing. When I think about testing, I look at an application in 3 broad layers: the code (on the machine side), the system (where the finished software lives), and the visible layer, or how the software is used from an end user’s perspective. I often call this visible layer the social context because of the environment much end-user software is used in. When we spend a lot of time in one context, testing starts to specialize because we concentrate on part of the picture.

When we view an application from the source code view, the testing is dominated by automation. When we look at the system context (as some of my operations friends do), testing involves integration in a system, and testing hardware, firmware, drivers, etc. to make sure the software gets served up correctly. Automated testing tends to get more complex the more we move from the code to the user interface. Attempting to emulate user actions is difficult, and high-volume automated functional tests can involve massive amounts of automated test code. This can be problematic to maintain. Sometimes functional tests become so complex they involve as much test code as the software they are testing has to serve up a component, plus test data generation code, as well as the code that attempts to emulate user actions.

Personally, I agree with Cem Kaner and call automated testing “computer-assisted testing”. The computer is a tool I use in conjunction with good manual testing. Until machines are intelligent, we can’t really automate testing. We can automate some aspects of testing to help maximize problem-solving efficiency.

Traditionally, software testers tend to have a handle on the social context, or how the software is used in a business context. As a result, much conventional testing is focused on the visible layer of the application. I tend to prefer testing at various layers in an application. I value testing components in isolation as well as testing software within a system. There are advantages and drawbacks to both. While I value isolation, testing, particularly brain-engaged manual testing at a visible UI layer has merit. Sometimes testers focus a great deal on testing in this context when component isolation might be more efficient. Often, traditional testers hope for an automated testing tool that can do the work of a tester. I’ve yet to see this occur successfully, but there are still tasks that can be automated to help testing efforts that are a big help.

Frequently there are bugs that are difficult to track down that crop up in the visible layer of the application. These are due to the visible application at runtime becoming greater than the sum of its parts. There is a kind of chaos theory situation that occurs due to the application being used in a way that the underlying code may not be designed to handle. By the time a minor fault at the code level bubbles up to the UI, it may have rippled through the application causing a catasrophic failure. Unfortunately, these kinds of usage-driven faults are problems automated tests at various layers do not tend to catch. Often it is some sort of strange timing issue as I note in this article. Other times, it’s due to actions undertaken in a social environment by an unpredictable, cognitive human. These variable actions motivated by inductive reasoning, driven by tacit knowledge are difficult to repeat by others.

Focusing too much on one testable interface in an application can skew our view. If we view the application by the code most of the time, we have a much different picture than if we view it through the UI. Try flipping the model of an application on it’s side instead of viewing it bottom up(from the code), or top down(from the ui). You may discover new areas to test that require different techniques, some of which are great candidates for automation, while others require manual, human testing.

Watir Release

The Web Testing with Ruby group released version 1.0 of the Watir tool today. Check it out here: Get Watir

Download the latest release which is a zip file at the top of the watir package list. (At the time of this posting, the latest version is 1.0.2.) Open the User Guide for installation instructions, and check out the examples to see how you can use the tool.

You will need Ruby if you don’t have it installed. Version 1.8.2-14 Final is a good candidate to use, or 1.8.1-12 if you prefer to be more on the trailing edge.

For those who may not be familiar with the tool, it allows for testing web applications at the web browser level. The project utilizes testable interfaces in web browsers to facilitate test automation. Currently it only supports Internet Explorer, but plans are underway to support other browsers. It is also only available on Windows.

Visible Tests and Web Testing with Ruby

Announcing a new project and blogging my notes for the Scripting Web Tests Tutorial which I prepared for XP Agile Universe:

Visible Tests

Automated test results are often difficult to communicate to business stakeholders on a project. People frequently ask me about the visibility of tests on a project saying: “The customer doesn’t understand JUnit tests” or “The JUnit and FIT tests mean very little to the customer. The customer doesn’t usually think in terms of green bars, or table-driven tests like developers and technical testers do. They believe we are testing, but they can’t see the tests run in a way that they relate to. How do we raise the visibility of tests?”

The WTR IE Controller has an advantage in test visibility since the testable interface we use is about as close as we can get to interacting with a program the way an end-user would. Business stakeholders understand testing at the GUI layer because that’s the way they relate to the program. These tests can be played back to the business stakeholders so they can see the tests run in a way they relate to. Technical people on a project team relate to the program at various layers and sometimes we forget about the business problems we are solving. Techies look at the backend and the front-end, business users usually only see and understand the front-end of the application. The preferred method of interaction with the program can differ between the groups.

Business stakeholders can watch the tests play back on a computer which provides rich, visual feedback. If the test fails, they see it fail. If it passes, they see it pass while exercising the application in a way that they would themselves. While they have faith in the ability of the technical members of the team, and will accept the testing numbers and their word, nothing replaces the assurance they get from seeing it work, and from manipulating the product themselves. With IE Controller tests, the customer can see if the tests pass or fail by watching how the application works in the Internet Explorer browser. Tests can also be designed to mimic business interaction, and provide results logging that non-technical project stakeholders can understand. The tests will demo the business solutions that the customer needs the application for in the first place. These kinds of automated tests help provide more assurance in what the technical members of the team are doing.

At an end of iteration meeting, the application can be demonstrated using these tests if the group wanted to give a quick demonstration.

Announcing WATIR

The latest version of the Web Testing with Ruby project has begun under the WATIR project spearheaded by Paul Rogers and Bret Pettichord. I’m excited by the prospect of a more sophisticated Open Source web testing solution using Ruby. I’ve had success with the IE Controller tests I’ve written to date, and look forward to using a more complete solution.

Test Automation as a Testing Tool

When we think of GUI-level test automation, we usually think of taking some sort of test case, designing a script with a language like Perl, Ruby or a vendor testing tool, and developing the test case programatically from beginning to end. The prevailing view is often that there is some sort of manual test case that we need to repeat, so we automate it in its entirety with a tool. This is a shallow view of test automation as James Bach, Bret Pettichord, Cem Kaner, Brian Marick and others have pointed out. I prefer the term “Computer Assisted Testing” (I believe coined by Cem Kaner) over “test automation” for this reason.

While developing automated tests, an interesting side effect came out in my development efforts. While I was debugging script code, I would run a portion of a test case many times. This would be a sequence of events; not an entire test case. I started to notice behavior changes from build to build when I watched a series of steps play back on my screen. When I would investigate, I found bugs that I may not have discovered doing pure manual testing, or running unattended automated tests. I started keeping snippets of scripts around to aid in exploratory testing activities, and found them very useful as another testing tool.

There are some benefits to automating a sequence of steps and blending this type of testing with manual testing. For example, if a test case has a large number of steps required to get to the area we want to focus testing on, using a script to automate that process helps us get there faster, frees us from distractions and helps us focus on the feature or unit under test. Humans get tired repeating tasks over and over and are prone to error. If precision is needed, computers can be programmed to help us. We can also test the software in a different way using a tool. We can easily control and vary test inputs, and measure what was different from previous test runs when bugs are discovered.

There are certain tasks that a human can do much better than a computer when it comes to testing. As James Bach says, testing is an interactive, cognitive process. Human reasoning and inference simply cannot be programmed into a test script, so the “test automation” notion will not replace a tester. Blending the investigative skills and natural curiousity of a tester with a tool that helps them discover bugs is a great way to focus some test automation efforts.

If you think of your automation efforts as “Computer Assisted Testing”, many more testing possibilities will come to mind. You just might harness technology in a more effective way, and it will show in your testing efforts.

I currently use Ruby as my Computer Assisted Testing tool, with the WTR Controller as my web application testing tool of choice. (cf. A Testing Win Using Ruby)