Category Archives: mobile testing

Book Excerpt: The Seven Deadly Sins of Mobile Apps

This is an excerpt from my book: “Tap Into Mobile Application Testing“, from Chapter 10, pp. 431-432:

Here is a simple summary I created to help you think of different areas an app can fail. This is another mind hack you can use to quickly organize your thoughts, and analyze an application quickly.

  1. Lust: the app advertises that it can do a lot more than it actually can. It leaves you feeling unfulfilled and wanting more.
  2. Gluttony: it uses far too many resources. It uses up your device memory, downloads large image and other files, and slows down your device, eats up your data plan and kills your battery.
  3. Greed: the assumes you have a strong network connection, and would love to use as much of your network resources as possible. The app can’t handle poor or weak connections, or transitions between network types when you’re on the move.
  4. Sloth: the app performs very poorly. It is far too slow to respond to your interactions, and takes too long to do anything useful.
  5. Wrath: it doesn’t play well with other apps. It has special settings that override your defaults, and doesn’t allow other services to work well with it. It may even cause other apps to malfunction because of its behavior.
  6. Envy: The app is too close to other available apps out there that you would prefer to use instead. It’s a copycat. You just wasted precious time and resources to get an app that wishes it was something else, but it just can’t deliver on its features, and usefulness.
  7. Pride: The app is difficult to use, expecting users to adapt to its way instead of helping you be more effective.You are subordinate to it, and you have trouble using it, and end up feeling stupid. This is most likely due to designers and developers assuming they know better than the users, and ignoring valuable usability feedback and usability bugs.

Thanks to Shannon Hale and Jared Quinert for their help with this list.

This is now a talk!

Note: I have created a one hour talk on this topic if your company, user group or conference are interested. I provide tips for designers, developers and testers on how to create apps that are engaging and reliable for mobile users.

 

Tap Into Mobile Application Testing Book Now Available in Beta

Update: The book is no longer in beta. The final version was published in September 2013 and is available here: Tap Into Mobile Application Testing.

How did you spend your summer? I spent mine writing a book: Tap Into Mobile Application Testing. Now that smartphones and tablets are taking over the mobile space, there has been tremendous demand for ideas and training on how to test on these devices. I started out with my Testing Mobile Apps course, but that only scales so far. Many of you have asked me about writing a book on the topic, so I did. It’s not completely finished yet, but I managed to create most of the content I had hoped to share.

I have two audiences with this book:

  1. Mobilists without much testing experience
  2. Professional testers without much mobile experience

I wrote the book with both of these audiences in mind, and I was also pleased to learn that even deeply experienced mobile testers told me they learned a lot of new ideas and approaches for their own testing projects after reviewing the book.

It is only available in electronic formats right now: PDF, epub and mobi, which should work on most e-reader apps on most mobile devices. I chose mobile first because that’s the medium many of my readers will choose first. It’s also faster for me to get content out there, and many traditional publishers print dead tree copies first, then look at electronic versions. I am using Leanpub, which allows me to publish free updates to every reader who has purchased a copy, even the finalized version that I am targeting for a release later in the year. Once I have a finalized version, I will provide print copies using a print-on-demand service.

The Beta version is a bit rough around the edges. I still have the following on my to-do list:

  • Copy editing and attribution clean up
  • Creating figures and images
  • Section rewrites and new material where needed

If you notice something wrong, misattributed, or missing feel free to contact me and I can fix it before the final version is complete in December.

If you are curious about mobile apps testing on modern devices, I hope you find it useful.

Creating a Test Lab for Mobile Apps Testing – Part 3

In parts one and two of this series, I introduced some ideas on building a physical test lab for testing mobile apps. In part 3, I’m going to talk about devices. This is all based off of my own experience.

Purchasing Devices

This is the the most difficult aspect of setting up a test lab. I’m not going to make any hard and fast recommendations, because by tomorrow the market will probably change. You will need to research and adjust depending on what your current needs are.

Emulators are fine for basic functional testing, but you’re going to need to do testing on the real thing, whether your focus is on automated or manual testing. The devices have sensor and touch interaction, as well as network and other physical aspects that need to have real-world testing.

You will need to consider the following:

  • both smartphones and tablets
  • different hardware and operating systems or versions, per framework
  • devices with different screen sizes
  • different wireless broadband carriers
  • cables and other accessories for charging and syncing (never hurts to have extras)

Each of these areas will yield different results when testing. For example, I frequently test with a smartphone and a tablet at the same time, and it’s amazing how the app can behave or appear differently on one or the other.

Also remember that a platform for testing is a combination of a hardware model, operating system version, and if you are using network connections with your app, wifi, wireless broadband, different carriers and any special software carriers install.

Purchasing devices is a balancing act, because the market is so dynamic. You will need to get a reasonable number of devices for testing the real thing, but you can’t buy them all. I look at the following to help guide what I should buy:

  • What platforms are our customers using?
  • What are the most popular smartphones and tablets on the market?
  • What are the most popular mobile browsers?
  • What platforms are known to be problematic? (aka. problem child devices)
  • When is the newest thing coming out? Do we need it for testing?

To figure out what you need to do using this information can be challenging. Here is one example. If you don’t know what your target customer base are using, you can start with information you already have. If you already have a web presence, or a web app, find out your web traffic stats. Are people using mobile browsers? What brand is the most popular? Can you get a visual breakdown? This information will usually breakdown according to brand, such as Android or Apple iOS, but it won’t give you specifics, such as an iPad, with 1st gen. hardware running iOS version 5.1.1. But, if you know that Apple iOS devices are the most popular, you can find out information about what the most popular devices are in the market, and assume that your customer breakdown is likely very similar. Look at a chart of popular versions, and try to replicate that in your lab – ie. stock up on the more popular devices, with less of the less popular devices. Not perfect, but it’s a good heuristic.

If you are looking at mass market applications, there is some information out there to help. For example, this chart this chart by Net Marketshare would indicate we should focus a good deal of our efforts with iOS device testing. Here is some more information to help deal with proportions of devices to purchase: comScore Reports May 2012 U.S. Mobile Subscriber Market Share. Akamai has an interesting report on mobile web browser breakdowns here: Akamai IO.

Notice that these reports are already out of date. This information is difficult to get, and I doubt anyone really knows exact numbers. It’s important to research and look at a lot of data to help inform you. When you spot trends, that should help you with proportions.

Some people think that Apple devices are much easier because there are fewer hardware devices, and they are all made by one company. However, even with Apple, you can end up with a lot of platform combinations. Take the iPhone for instance. Let’s say we are going to have 3 devices: an iPhone 3GS, an iPhone 4, and an iPhone 4S. Since Apple encourages people to upgrade their OS, we have decided to stick with iOS 5, the latest version. At the time of writing, there are 4 iOS versions out there: iOS 5, iOS 5.0.1, iOS 5.1.1. Also assume our app requires an internet connection. If we choose 3G, the most popular wireless broadband networking type, and wifi, then we have 24 platform combinations. What if we now need to test on 4G now? Add 12 to the mix for a total of 36. That can be a lot of regression testing. If you want to optimize, you may decide to elminate everything but the latest version, 5.1.1. If you do that, try to find something that gives you an indication of upgrade proportions, like this information: iOS 5.1.1 Upgrade Stats. You may not want to upgrade all of your devices at once, but keep some back to mimic what is happening in the real world.

Here is a recent plan for an iOS test lab:
Apple iOS (all on the latest, 5.1.1):

  • (1) iPhone 3GS
  • (2) iPhone 4
  • (3) iPhone 4S
  • (1) iPod touch (they aren’t as powerful, and are great for finding problems you don’t see on other devices)
  • (1) iPad first gen
  • (2) iPad second gen
  • (3) iPad third gen
  • (at least 1) whatever the newest iOS device is when it comes out(these are always hugely popular)

Android, Windows Phone and Blackberry are more difficult, not just because of the different handset manufacturers, but there is more of a proportion of devices across versions of their OS, or OS family. For example, Android has a nice graphic of their Android OS Distribution. When setting up a lab for Android devices, I would try to replicate that pie chart in my own device distribution. For older platform families like Windows and Blackberry, you will have even more variations to consider, but temper that with how popular they are in your target market.

For mass market applications, or web testing, Brad Frost has a fabulous blog post test on real mobile devices without breaking the bank, and this blog post from the Mobile in Higher Ed blog is also very useful.

It sounds expensive, and it can be, but compared to test workstations and servers, it isn’t that bad. You can get multiple test devices for the price of many test servers when you take hardware, OS and required software into account. You will have to research using your favorite search engine, and you’ll need to stay on top of new developments, because you will regularly need to acquire new devices.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.

Creating a Test Lab for Mobile Apps Testing – Part 2

In Part 1 of this series, I introduced some ideas on building a physical test lab for testing mobile apps. In part 2, I expand on those ideas.

Storage

You will want a clean, dry, secure storage area for devices to be kept in when not in use. I’ve used a drawer that locks with toolbox liner at the bottom to protect the cases and keep them from moving around. Here is an example: toolbox liner. The devices are small and disappear easily, so secure storage is a must.

Set up a sign-out sheet with each device noted by OS, version and serial number. Also create tags for each power/sync cable for each device so they can be kept track of if they leave the test lab. I used a label maker for this. There is nothing worse than losing your hard-earned test devices to other groups, or not having devices around when you need them. Cables disappear the most, so you need to make ownership for your lab visible. Think of the washroom key for a service station on the side of a busy road – the small key is often connected to something larger to make it more visible to help prevent it from getting lost. I always chuckle when I see a key on a chain with a hubcap attached, and I try to make my lab cables visible in some easily identifiable way.

Power Station

You will need access to power to charge the devices so they are available for testing when you are. It’s incredibly frustrating when you are getting ready to test on a device and find out that the battery is dead. If you can’t get power into your device storage area, set up a charging station in the lab.

Set aside counter or desk space near a power source, and add the following:

  • at least one power bar to plug in multiple devices
  • several power cables for the devices
  • toolbox liner or something similar on the surface to keep them from moving around

Lab PCs

You will need computers to manage the devices for testing, troubleshooting and other tasks. Consider at least one machine for each platform family. Here are four popular platforms. Your team will have their own mix of devices that they are developing for. Get the most powerful workstation you can afford for each platform; mobile app development tools can be large and complex.

iOS (Apple)

You will need at least one Mac for emulators and device management, and pulling screenshots and other information from the devices. Install XCode for emulator use, Also consider installing the Network Link Conditioner to help simulate different network connection conditions with an emulator. Install the iPhone Configuration Utility for getting stack traces, managing licenses, etc.

Android

You will need at least one machine with the Android SDK installed for managing Android devices and emulator use. Many developers use this within the Eclipse IDE, so talk to them about the correct combination of tools for your shop. The operating system doesn’t matter for Android development.

Windows Phone

If you are supporting Windows Phones, you will need a Windows machine with at least Windows 7 and the Windows Phone SDK and related tools.

BlackBerry

The new BlackBerry SDK supports several languages and implementations, so you’ll need to find out which version to install, and what other libraries need to be installed. I don’t think OS matters, but I have seen installation instructions recommending at least Windows XP or Vista.

Wireless Broadband

Data plans or other contracts for monthly wireless broadband access are important to test. I would get a reasonable plan for each device, from at least two local telecom providers. Try a larger provider, and a smaller provider (like a discount provider) to get more variation in wireless broadband technology. There is a huge diversity in what different carriers use to deliver this to the end user, so try to get some variation

Other

You will need internal test email addresses, and phone numbers for each smartphone for testing. Also get at least one developer license for each platform your team supports for your test lab. You will need test ids with the App Store on Apple for making purchases, or installing apps from the app store. You will likely need to do this with other stores as well such as Google Play and others for each device type you have.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.

Creating a Test Lab for Mobile Apps Testing – Part 1

Some of you have asked me to expand on the physical aspects of mobile app development that I described in my article: Mobile Challenges for Project Management. If we have a testing burden when we develop mobile apps, what sort of considerations should we take into account when we are setting up test labs, when using real devices? Here are some pointers from my own experience.

This is part 1 of a 3 part series.  Feel free to add comments for anything I’ve missed.

The Room

You’ll need space where you can set up PCs for managing devices and using productivity tools, a place to store devices when they aren’t in use, space for charging devices, and plenty of space for people to move around with the devices while testing. (The mobility side of mobile devices needs to be tested.)

Network

Wifi

I prefer to have at least two dedicated test wi-fi beacons to test network connections and transitions between wifi, and wifi to wireless broadband. I prefer small, cheaper devices that have weaker strength so it is easier to set up test conditions for apps that require network connections. I like to have these for test purposes only, and it is better that they have weaker signals (so testers can walk away and get weaker wifi for testing), and can be turned off or unplugged for testing outages, etc. Also since they are smaller and weaker, you can set up two in a lab and transition between the two by moving in the lab.

Simulate Dead Spots

If you don’t have an office dead spot (no wireless broadband or wifi connectivity), or one that is convenient for testing, you may need to set something up yourself. If you have an elevator or other known dead spot near your office, that works well. In some buildings you don’t.

Consider building a faraday cage to test for dead spots. Try doing network/web searches or submissions within a deadspot, or transitioning into or out of one. You can use an old microwave, but make sure you cut off the power cord so it can’t ever be powered on accidentally. They are simple to use, try a web search or submission, or any action that requires a network connection, and put the device into the microwave. It will instantly be in a dead spot.

You can also buy them from providers, or make one out of wood and wire mesh:

  • build a box out of 2×4’s, hinges and wire mesh
  • create a door in the front
  • wrap the box with wire or brass mesh
  • attach a ground wire to the box

There are different instructions online on how to make them.

To test outages, or performance issues, and to monitor and measure performance, page sizes, security, and other things,  you may want your own network for testing. It’s a good idea to have a separate test network, so you can quickly create scenarios, outages, or easily monitor mobile app traffic, whether it is a physically separate network, or a virtual network without interrupting other work.  Your administrators can figure out the implementation.

Stay tuned for parts 2 and 3.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.

Public Course: Testing Mobile Applications, in London, UK, May 10-11

Working with Electromind, I’m offering a 2 day public course on Testing Mobile Applications in London, UK, May 10-11, 2012.

Register here: Testing Mobile Apps Registration

Testing Mobile Applications Workshop Description

2 day tutorial by Jonathan Kohl.

Applications for smartphones and tablets are incredibly popular. Teams are facing pressure to deliver testing quickly and successfully on mobile devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. If you follow the same practices and techniques as you have before, you will miss critical bugs. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Hear first-hand experiences with testing mobile applications and discuss how to address various challenges.

Note: Participants must bring their own mobile device for course exercises. This is a hands-on course.

Public Workshop: Testing Mobile Applications, in Vancouver, May 25

As part of the Agile Vancouver Quality in Agile conference, I’m offering a 1 day public workshop on Testing Mobile Applications in Vancouver, BC, Canada, May 25, 2012.

I will also be speaking about testing and value during the conference on May 26.

Register here: Quality in Agile Conference Registration

Testing Mobile Applications Workshop Description

1 day tutorial by Jonathan Kohl.

Applications for smartphones and tablets are incredibly popular. Teams are facing pressure to deliver testing quickly and successfully on mobile devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. If you follow the same practices and techniques as you have before, you will miss critical bugs. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Hear first-hand experiences with testing mobile applications and discuss how to address various challenges.

Note: Participants must bring their own mobile device for course exercises. This is a hands-on course.