Category Archives: models

Speeding Up Observation with Models

You just generated four possible ways to solve the problem in about thirty seconds!

One of my brilliant programmer friends had just posed one of those useless interviewing puzzles that Microsoft made famous. We were discussing that the only thing they test is a candidate’s ability to solve those sorts of problems, and really tell you nothing about their ability to do software development tasks. He was frustrated by a recent interviewer who had merely posed a puzzle, wrinkled up their nose at his attempt to solve it in the interview, and had then lectured him on the “right” answer. To illustrate, he had posed the question to me.

Fortunately, it was a type of puzzle that I could easily apply a model against. Also in my favor, I had spent a recent evening teaching a software testing student that very model. I smiled inwardly, and with nonchalance outlined four possible ways of attacking the problem. He stared at me almost dumb-founded for a moment, and then I had to let him off the hook. I laughed as he stared at me in wonder. “There’s a trick to this” I told him. I revealed the model, which is the software testing mnemonic HICCUPPS. I suddenly had a brilliant programmer with a much stronger intellect than mine very interested in software testing. “Teach it to me!” he chortled, with relief and delight.

We both agreed that the “right” answer the interviewer posed was a complete joke, and he was justified in concluding that this was not a company he wanted to work for. That company lost out on hiring one of the best problem-solvers I know.

“Teach me more about these testing mnemonics!” he admonished as we parted ways that day.

* * *

How did you generate so much thorough analysis so quickly? I’ve been here for three weeks, and you’ve been here for three days!

I was doing analysis work with a friend of mine, who like my friend above, is also a brilliant programmer. When I was still in short pants, he was creating artificial intelligence programs in Lisp.

“Part of the reason is that I have your three weeks of work to draw on.” I replied. “Of course, but you have touched on areas I hadn’t thought about at all. I would have thought of them eventually, but how did you think of them so quickly?” he said with a questioning grin. “I use models”, I replied, and brought out a testing problem to demonstrate my thinking. He grasped the object I use in that problem for a second, then looked at me, handed it back and said: “Oh, I see how you do it. Raw observation is very slow. Your models help frame your thinking and speed up observation.” He didn’t even try to solve the problem, and in his characteristic way, rapidly understood a lesson that had taken me much longer to learn.

“Good work” he said with a chuckle as he got up to leave for home. My speech on the wonders of James Bach’s San Francisco Depot (SFDPOT) mnemonic would have to wait for another day.

* * *
Models are useful for generating ideas when solving problems. They aren’t fool-proof though. I am careful to not ignore, but to absorb and test my first instinctive thoughts when solving a problem. Often, that intuition helps, but sometimes I discard it, as careful scrutiny denies the plausibility of the initial thought impulse. Intuition can be both powerful and perilous. There’s nothing wrong with intuition, as long as you can make your conclusions defensible.

It doesn’t hurt to complement intuitive thought with a model, and I find using both helps kick start my thinking processes. Sometimes, the most lucid and effective ideas sprout out of nowhere from a sea of wonder and half-baked thoughts. They frequently pop into my head as I am using a model, and they often turn out to be unrelated to my current train of thought, but incredibly useful. I don’t ignore them merely because I can’t explain them. Instead, I try to adapt my models to increase the chances of them occurring.

Models also add consistency when solving problems. Instead of a formulaic, step-by-step, “one size fits all” approach to solve a problem, they can be used to consistently remind our brains to look into diverse areas for clues and impulses. This is much more powerful than a rote process that may, or as in most cases, may not work. It greatly increases our chances of coming up with a unique solution for a unique problem. I find that after a considerable degree of rumination, I have an “aha” moment, but using a model to complement my natural thinking usually brings me to an “aha – why didn’t I think of that before?!!” thought much more quickly.

Models also help build confidence. If I am posed with a difficult problem, and I have no idea where to start, I can nervously waver and pause, or I can jump in with confidence and use a model. It doesn’t matter if it’s the wrong one, the thinking drives out the fear and uncertainty and leads me in a direction towards solving the problem. Fear steals energy, while models give me a place to start in the face of uncertainty. I constantly learn from experience, and develop new models when I learn a new lesson. Variety of models makes the problem-solving space much richer, and less intimidating.

A good friend of mine is a firefighter, and he also uses mnemonics in his work. His son had recently taken an emergency services technician course, and he was drilling his knowledge one evening. They both have been taught mnemonics to help frame their thinking in emergency situations. In fact, they and others have told me that attempts at scripting this kind of problem solving ends up in farcical situations, or worse, loss of life.

That sobering thought aside, it was a pleasant evening together, and I sat amazed as they rattled off one mnemonic after the other. They even had mnemonics within mnemonics. While one would set the scene for a new problem, the other would rattle off a mnemonic, and explain how they would use it to observe something different to help them get a fuller picture of the problem. They are terrified of having narrow observation within the small window of time they are usually faced with. “But now you notice this in your patient. What do you do now?” From within one letter of one mnemonic, the other would rattle off another mnemonic to use to help jog their observation and thinking to solve that problem, with speed, coherence, and confidence. They would also frequently throw the mnemonics aside altogether, and use other heuristic-based thinking techniques.

They were both in the heat of the discussion, each trying to trip the other up, and the father was attempting to pass on wisdom and experience to his son in a curious Socratic style. I chuckled to myself. I felt if at that instant I were to keel over with some health-related malady, I would be in very capable hands, and someone would learn a new thinking technique in the process.

RE: Model Myopia in Testing

Chris McMahon asks:

How do I find a useful model when unit tests are passing and show 100% code coverage?

Chris asked me this a bit tongue-in-cheek because he has heard me rant about these sorts of things before. I’m glad he asked the question though, because it brings up a different kind of model myopia. In yesterday’s post, we have modeling myopia with regards to the application we are testing. This question brings up model myopia regarding our potential testing contexts.

This is a common question, particularly on agile teams that utilize automated unit testing who are struggling with a tricky bug. Let’s examine this situation in more detail. What does the claim in the question tell us about testing?

This claim provides information about a certain testing context, the code context. We know that within this testing context, we have a measure of coverage that claims 100%, and that the tests that achieve this coverage claim pass. Is this enough information? We have a quantitative claim, but this doesn’t tell us much from a qualitative perspective.

Why does this matter?

One project I saw had 300 automated unit tests that achieved a high level of code coverage on a small application. When we did testing in another context (the user context, or through a Graphical User Interface), the application couldn’t pass basic tests. On further investigation, we found that most of the unit tests were programmed to pass, no matter what the result of the test was. The developers were measured on the amount of automated unit tests they wrote, on the code coverage attained as reported by a coverage tool, and they could only check in code when the tests passed. As a measurement maxim says: show me how people are measured, and I’ll show you how they behave.

This is an extreme example, but underlines how dangerous numbers can be without a context. Sure, we had a number of passing tests with a high degree of code coverage on a simple application, but the numbers were almost meaningless from a management perspective because the tests were of poor quality, and many of them didn’t really test anything. I’ve seen reams of automated test code not test anything too many times to take a quantitative claim on its own. (I’ve also seen a lot of manual tests that didn’t test anything much either.)

Now what happens if we agree that the quality of the unit tests are sufficiently good? We don’t have ridiculous measures like SMART goals where we “pay for performance” on automated unit tests, code coverage, bug counts, etc., so we don’t have this “meet the numbers” mentality that frustrates our goals. We know that our tests are well vetted, thought out and have helped steer our design. Our developers are intrinsically motivated to write solid tests, and they are talented and trustworthy. What now? All the tests passed. Where can the problem be?

It’s important to understand that “all the tests we could think of at a particular time passed”, not that “all the possible tests we could ever think of, express in program code and run have passed”. This is an important distinction. Brian Marick has good stories about faults of ommission, and reminds us that a test is only as good as what it tests. It doesn’t test what we didn’t write in the first place. Narrow modeling is usually a culprit, and once we learn more about the problem we can write a test for it. It’s hard to try to write a test for a particular problem that has stumped us and we don’t have enough information yet, or to predict all possible problems up front when doing TDD.

In other cases, I have seen tests pass in an automated framework that were due to using the framework incorrectly which caused false positives. This is rare, but I have seen it happen.

Michael Bolton has summed up automated unit testing assumptions eloquently:

When someone says “all the unit tests passed”, we should hear “the unit tests that we thought of, for theories of error of which we were aware, and that we considered important enough to write given the time we had, and that we presume to have run on the units of the product that we assume to have been built, and that we presume actually test something, and that we presume provide a trustworthy result 100% of the time, passed.”

Testing Contexts

Another important distinction to make is that we know this information comes from one testing context, the code context. We know there are others. I split an application with a user interface into three broad contexts: the code, the system the application is installed in, and the user interface. We know at the user interface layer, the application is greater than the sum of its parts. It will often react differently when tested from this context than it will from the code context. There are many potential testable interfaces within each context that can provide different information when testing. What interfaces, and in what contexts are we gathering test-generated information from?

When I see a claim like this: “we have 100% of our automated unit tests passing with 100% code coverage”, that only tells me about one of three possible model categories for testing. If we haven’t done testing in other contexts, we are only seeing part of the picture. We might be falling into a different type of modeling myopia – looking too narrowly at one testing execution model.

It’s common to only view the application through the context we are used to. I tend to use the term “interface within a context” rather than “black box” and “white box” after my experience with TDD. The interface we interact with the software with the most can cause us to narrow our focus on testing. We then model the testing we can do with a preference to this interface. Mike Kelly has a blog post that deals with bounded awareness and inattentional blindness, and how they can affect our decisions.

I see this frequently with developers who are working predominantly in the code context, and testers who are testing only in the user interface context. Frequently, each treat testing in the other context with some disdain. Often, testing diversity pays off, and when we work together bringing expertise and ideas from any testable interface we were using, we can combat model myopia.

Testing on one context does not mean testing in another context is duplicated effort or unnecessary. Often, testing using a different model reveals a bug that isn’t obvious, or easily reproducible in one context.

Not too long ago, I had two meetings. One was with two developers who asked why I bothered to test against the UI when we had so many other interfaces that were easier to test against behind the GUI. In another meeting, a tester asked why I bothered doing any testing behind the GUI when the end user only uses the GUI anyway. In both cases, my answer was the same: “When I stop finding bugs that you appreciate and find useful in one interface that I couldn’t find more easily in the other, I’ll stop testing against that interface.”

When testing in a context fails to provide us with useful information about the product we are testing, we can drop that model in its importance to our testing. Knowing about a testing model and choosing not to use it is different than thinking we have covered all ourtesting models, and still be stumped as to why we have problems.

Combating Model Myopia in Testing

Part 1

In my last post, I posed a testing problem. This post begins to address how I view the problem, and my motivation behind posing it.

Models

In software testing, a “model” is a representation of something we are trying to understand. It might be a representation in our mind (an idea), a diagram (like an Entity-Relationship diagram), or it might be a physical representation. A model helps us think of a system or concept in a certain way, and as a representation is not going to be perfect or complete. We can have many models for the same thing – our only limitation is our imagination.

Here’s an example:

How many ways can you model a web page? I can think of at least three categories:

  1. as an end-user, or consumer view where I am using a web page for some purpose. I see the webpage as having text, colors, pictures and hyperlinks. I visualize that web page, served up in a web browser when I think about a particular page I find useful, and want to go to it. I interface with it and relate to it as it is shown through a web browser.
  2. as a web developer. I tend to develop web pages using a text editor and do all of the HTML, CSS, etc. by hand. I interface with the web page with a text editor, and I visualize it and relate to it by the HTML view which is made up of markup tags. I interface with the web page primarily through a text editor.
  3. as a web designer. I either have a visual picture in my mind of the layout and design of a web site I want to create and I follow that, or sometimes I draw it out using a tool. Even a rough sketch of a general layout on a piece of paper or on a whiteboard will convey that idea. I visualize the web page according to that layout.

These are three simple examples of modeling a web page. I added a couple of dimensions to help describe the models – a user, and an interface. Each one is different, but I may switch between all three as I’m developing a web page, or when I’m testing.

Since I often test web applications, I think of these different views a lot. I think of the first two very often, especially since I did a lot of web development in the mid-late ’90s. When I started writing web applications, I also learned to model from a programming context. My HTML tags, page layout, and content would be generated according to program logic, so I had a fourth model category derived from my role as a web application programmer. Understanding all the components used in rendering a web page is important to me when testing a web application. Everything from the web browser to the application server, web server, database if one is used, any drivers that are used, as well as the program code design is important information that can help with testing. I actually visualize what is happening across the entire application path when data is input into a web page, sent to the web server, processed, and the page I am on is updated. Even a simple HTML page has an interesting path. From the web server we have a text document that contains HTML markup, with certain tags that denote it as such. When a web browser asks for that page, the web server takes the tags, processes them and sends them via HTTP to the machine that is asking for it. The web browser takes that HTTP response, interprets it, and if it comes through without an error, it processes the HTML and displays it in a web browser.

Why is this important?

Modeling applications like this helps me imagine areas of potential problems based on experience and technical knowledge of the various technologies used, and also helps me narrow down or get repeatable cases for bugs I find in the application. It also helps me generate different testing ideas instead of one “happy path” based off of requirements documents, I can think of other potential usage of the software that might have the potential for problems. When I work with software users in their own environment, I am always amazed at how they use the software in unintended ways which they are perfectly happy doing.

What other models can you think of when testing web applications? Who are the users? Not all of them will be human. What are the interfaces? Not all of them will involve a Graphical User Interface like most modern, familiar web browsers.

But I’ve Run Out of Ideas!

The problem I described was something I deliberately chose as a modeling problem. Notice that in the energy industry, when I got a failure on a search attempt that contained the word “well”, I immediately had a suspicion of what the problem might be. But if I was in another industry, “well” may not be in my common vocabulary as a noun, but primarily an adverb. (Note that language is important when testing.) In my description here, we have at least two modeling categories of the software. One as “enterprise search”, and one as “my energy company search”. The ideas that each model represent are going to be subtly different. That subtle shift between models can be the difference between tracking down the source of a bug or not. But if I didn’t have energy experience, how can I find that model that will help me track down the bug?

I call this so-called running out of ideas “testing model myopia”. If we were omniscient, we would know all the possible ways to model the software, and we’d know where all the bugs were. If we were semi-omniscient and merely knew all the possible ways to model our software, we’d have an easier time. Since we aren’t, we have to use our skill as testing thinkers to break out of mental blocks. This can be very difficult if the entire team has testing model myopia as I described in the problem. When under project stress, with constraints on a development team such as time and resources, a tester may have to build a case in order to be able to pursue a testing idea. The predominant model of the software as shared by the team can be very entrenched, and it can be threatening to have that model questioned or analyzed. Nevertheless, as a tester, we are what Jerry Weinberg has described: “someone who can imagine that there might be another way.”

One of the first places I would start to tackle this problem is to imagine other models of the software that we aren’t addressing when testing.

Stay tuned for Part 2.

Testing an Application in Layers

There is often debate about test automation versus manual testing. When I think about testing, I look at an application in 3 broad layers: the code (on the machine side), the system (where the finished software lives), and the visible layer, or how the software is used from an end user’s perspective. I often call this visible layer the social context because of the environment much end-user software is used in. When we spend a lot of time in one context, testing starts to specialize because we concentrate on part of the picture.

When we view an application from the source code view, the testing is dominated by automation. When we look at the system context (as some of my operations friends do), testing involves integration in a system, and testing hardware, firmware, drivers, etc. to make sure the software gets served up correctly. Automated testing tends to get more complex the more we move from the code to the user interface. Attempting to emulate user actions is difficult, and high-volume automated functional tests can involve massive amounts of automated test code. This can be problematic to maintain. Sometimes functional tests become so complex they involve as much test code as the software they are testing has to serve up a component, plus test data generation code, as well as the code that attempts to emulate user actions.

Personally, I agree with Cem Kaner and call automated testing “computer-assisted testing”. The computer is a tool I use in conjunction with good manual testing. Until machines are intelligent, we can’t really automate testing. We can automate some aspects of testing to help maximize problem-solving efficiency.

Traditionally, software testers tend to have a handle on the social context, or how the software is used in a business context. As a result, much conventional testing is focused on the visible layer of the application. I tend to prefer testing at various layers in an application. I value testing components in isolation as well as testing software within a system. There are advantages and drawbacks to both. While I value isolation, testing, particularly brain-engaged manual testing at a visible UI layer has merit. Sometimes testers focus a great deal on testing in this context when component isolation might be more efficient. Often, traditional testers hope for an automated testing tool that can do the work of a tester. I’ve yet to see this occur successfully, but there are still tasks that can be automated to help testing efforts that are a big help.

Frequently there are bugs that are difficult to track down that crop up in the visible layer of the application. These are due to the visible application at runtime becoming greater than the sum of its parts. There is a kind of chaos theory situation that occurs due to the application being used in a way that the underlying code may not be designed to handle. By the time a minor fault at the code level bubbles up to the UI, it may have rippled through the application causing a catasrophic failure. Unfortunately, these kinds of usage-driven faults are problems automated tests at various layers do not tend to catch. Often it is some sort of strange timing issue as I note in this article. Other times, it’s due to actions undertaken in a social environment by an unpredictable, cognitive human. These variable actions motivated by inductive reasoning, driven by tacit knowledge are difficult to repeat by others.

Focusing too much on one testable interface in an application can skew our view. If we view the application by the code most of the time, we have a much different picture than if we view it through the UI. Try flipping the model of an application on it’s side instead of viewing it bottom up(from the code), or top down(from the ui). You may discover new areas to test that require different techniques, some of which are great candidates for automation, while others require manual, human testing.

Tester as Editor Metaphor

Sometimes when I’m asked to explain how I see the role of testers on development teams, I use an (admittedly simplified) editor role as an analogy. For example, I might be asked to write an article on a particular topic, say test automation, and the publisher asks me to touch on three areas: a test strategy, a list of test tools and an example. These three requirements serve as the acceptance tests. I will write several drafts on my own, and with each draft I will remove spelling errors and grammatical mistakes. These are my unit tests. I move iteratively with article drafts until I feel I have expressed the ideas that are needed for the article. Once my unit tests have passed, and I’m confident I’ve have touched on the three areas I’ve been asked to write about (my acceptance tests seem to pass), I send the article to an editor for feedback.

The editor checks the content. If I haven’t met the obligation of the acceptance tests, they will let me know quickly that I missed a requirement. Good editors point out areas where I haven’t expressed an idea as clearly as I could. Good editors really know the audience, and point out areas that the audience may have problems with. They may also point out spelling mistakes and grammatical errors I have missed. They help draw out ideas and help me make the article all it could be given the time and resources available.

The editor doesn’t merely point out flaws, they can also provide suggestions to help overcome them. A cluster of writing errors may indicate to an author that an idea is malformed. Editors can spot problems that the author might miss because they are a skilled “second set of eyes”. They can provide constructive criticism and help encourage the author prior to publishing. It isn’t necessary for all articles to have a formal editor, but when I work with a good editor I realize how much better the article is than if I did it on my own.

In many ways a software program is also an expression of an idea. An idea may be technically correct and may meet the requirements of the customer, but may not be expressed clearly given the intended audience. My role of a tester is not an adversarial one, but like the editor my goal is to encourage and help make the program all it could be prior to its release. Like a good editor, a tester has an ability to know the audience of the program and the context in which it will operate.

In this post, Brian Marick talks about writer’s workshops which look like a good idea for doing software testing:

I’m drawn to the analogy of writers’ workshops as they are used in the patterns community. Considerable care is taken to prepare the author to be receptive to critical comments. At its worst, that leads to nicey-nice self-censorship of criticism. But at its best, it allows the author to really hear and reflect on critical ideas, freed of the need to be defensive, in a format where teaching what should change doesn’t overwhelm teaching what works and should stay the same.

Software development teams could learn a lot about constructive criticism from the good writer’s workshops.
In another post, Brian sums up the idea I’m trying to get across here:

… I offer this picture of an Agile team. They – programmers, testers, business experts – are in the business of protecting and nurturing the growing work until it’s ready to face the world.

Good testers are not the adversaries of developers, they are part of the team who works collectively towards creating the best software they can. Bad testing, like bad editing does not seem to be in the business of nurturing a growing work.