Mobile UX Testing

Five Considerations for Mobile UX Testing

It’s no secret that mobile user testing is just as important as testing your web app. In fact, it may be even more critical to your success, especially if you have a heavily reliant on mobile users. Think: Lyft or Instagram which are almost entirely on mobile.

Whether you have a native apps, mobile web app, or hybrid app, you’ll need to run similar user research studies to ensure what you ship achieves the desired outcomes.
Let’s highlight a few differences in mobile user testing versus desktop and what you need to consider:
  1. Mobile is.. well, mobile! Your user is unlikely to be sitting as a desk in a chair. They could be on the bus, walking down the street, or even, yes, in the bathroom. Consideration about when, where and how a user navigates your mobile app can help set the right context. Your mobile app may be specific for location (think Google Maps, or a GPS athletic tracker such as Strava).

    Consider: Have you thought about where your users may be, or you need them to be, for accurate results? If you’re testing in a lab, you may be OK with low-priority issues, however, you may miss the context for high-severity issues.

  2. It’s all about context, and yes the possibility of a user sitting on the throne in the bathroom. Privacy is a greater concern when testing on mobile.

    Consider: What kind of data are you ingesting? Are you capturing their location in addition to screencast and video? What implications might any of those have on your test? Might you see other data such as personal photographs as they navigate in between apps or other data on their phone that may otherwise be private?

  3. Shorter interactions are likely. Not necessarily in duration, but regardless of how large that phablet is, it’s still not a laptop nor a desktop, so the content will need to be fitted appropriately.

    Consider: What specific parts of your test have you designed that allow for shorter interactions.

  4. Interruptions. Your users are likely to be interrupted. They might receive a text message, or some other app’s notification while they are undergoing your test. The text from their mother is probably far more important, so expect that your test might get put on hold.

    Consider: Have you thought about how to structure your test so it’s easy to come back to?

  5. Non-linear flow. A mobile experience is not a single user experience as there are often many pathways a user takes to accomplish a task. Consider: Testing multiple pathways will help you develop an application that fits into your users life.
Do you have other mobile considerations? Drop us a note!
Finding Time for UX Research

There’s Never Time for User Research

Have you heard these before? We have. Here are some excuses on why people avoid user research.

It takes too long

There are many methods that can be used for user research. Some of them, like a multi-country ethnographic study, will take long. But there are many other methods that can be carried out in weeks, days, even hours. Don’t be pennywise and pound foolish, a few hours or even days to get it right is a very high ROI. Shipping the wrong feature fast provides no value to your users.

There’s no budget

This is similar to the It takes too long argument. An international survey with 20,000 participants followed by in-person interviews will cost a lot. But that is a very rare project. A majority of your research needs we observe in our modern work environment can be addressed with budget-friendly methods, iteratively, without compromising quality. Ask yourself: do you have the budget to design and develop your product twice?

We are not researchers

We’ll let you in on a secret. Once upon a time, none of us were researchers! Research, like any other discipline, is a learnable skill. Your title does not have to be researcher to do research, and you don’t need a science or a design degree.

We don’t have a usability lab

It’s good that you do not have a usability lab, spend your budget elsewhere! It is better that you go to your users and become a part of their daily life to understand their usage within context, instead of a glass room.
If your research question calls for these, yes you need a usability lab. But those questions are not the most common research questions we see.

This is completely new

The team is excited about their new, from-scratch project. They feel that they do not need to do user research, because this is completely new and novel. That there is nothing to compare it against. That could be true, but there are ways to gather feedback from your potential users even if the product is completely new. Also, you are taking on significant risk if your new product meets the users for the first time at the time of production. Getting early feedback and making changes while you can creates a much better product.

But it’s just a small change

Many small changes accumulate over time to create big changes. It is great that your organization can ship small changes and ship them quickly, but that does not change the fact that you are serving a new experience to your users. Small changes can and should be validated through user testing.
There are more for sure, I’m sure you have some you could add to this. Remove the ego from your product efforts and let the evidence from your research guide your product.
How you interact with your customers shows how successful you'll be

How you interact with your customers shows how successful you’ll be

When a colleague of mine and I were visiting a customer some time ago, our initial intent for the meeting was to resolve a few specific issues they were having with our product. As we reached onto a particular topic, his next sentence was “wouldn’t it be really cool if you had…” I don’t need to finish the specifics of that sentence. You can fill it in with just about any idea. This type of question was all about confirming his initial idea. Further, the question was quite leading. It turns out there are thee basic ways you can interact with a customer.

Transactional: When you’re a hammer, everything looks like a nail. This approach is exactly what it sounds like, it’s all about the transaction. you might hear a question such as: “Would you like to buy…”

Confirmatory: You may have heard a statistician say “If you beat up data it will tell you what you want to hear.” This approach is about confirming biases that one already has and not truly listening to what the customer has to say. You might hear questions that start with “Wouldn’t it be cool if you had…” “What if you had..”

Diagnostic: Let’s get to the real problem! This approach is all about trying to understand the needs of the customer with as little bias as possible and being open to hearing “Tell me about the last time you..” or “What happend when…” These are often based on concrete past events and decisions, not a hypothetical future.

A friend who runs a startup accelerator remarked that of all the startups he sees, the most successful are the ones who take a the diagnostic approach to understanding their customers. Which approach have you been following? I’ll admit to being in the confirmatory approach in my early product years, then I realized I was just looking to be right, and all that did was make my product more wrong.

It's important to pick the right people to test.

Recruiting the Right Test Participants

When your faucet is leaky, you don’t call an electrician because you need someone who has the right expertise to help you. So let’s say you work for a health record (EHR) software company. and you need to do some user testing. You certainly wouldn’t call up your accountant and ask if she was available for a short test. While it’s tempting to gather feedback from a user group that takes little effort to access (and I’m not saying all accountants are rapidly accessible), you need to ensure you are testing with the right target audience to get the highest quality insights.

When I led an innovation team we struggled with this, so our team built out a database of customers and non-customers that we could run tests on when we needed them. Ultimately we had about 10,000 participants who all opted into this program. When we needed to run a test with 10 to 20 people, we would search this database for the right targets. It saved significant recruitment time and cost. However, it’s not always the case and that took a good 6 months to build out.

So what do you do?

First, identify the right participants.

One team I spoke with a few years ago needed to find athletic coaches for endurance athletes as their app had a specific use-case and target audience. Heading to the local Starbucks or even posting an ad to Craigslist may not quite work. For some consumer apps sourcing on Craigslist may actually work though you’ll need to use a screener questionnaire. It’ slikely you already know this as when you were creating your designs for your product you probably had some personas you designed for.

Second, go where they are.

In the example above example: where do endurance coaches hang out? Is there a secret endurance coach club you can visit? No! But there are running and cycling races, as well as forums, so finding your way there might help you learn who may be a good candidate recruit to your test.

Third, screen for quality.

When you have a clear view of who you need to test, you’ll need to create a short screener question to weed out the incorrect participants. These can include simple questions like: “how many athletes do you coach on a regular basis?” Sure there may be demographic questions, however if you can get to question wich selects for the behavior type,this can increase the quality of participant you test with.

Lastly, find an emotional incentive.

Gift cards are all well and good, but if there’s a better incentive you can offer your test participant, that will make a difference in the quality. I lost count of how many people I tested with over the years who up front asked me if I had the $20 gift card before the test even started. This indicated to me it was all about an economic transaction and not necessarily a quality information exchange. Finding customers who have a distinct problem you’re solving for so the benefit to them isn’t a free lunch, rather it’s a problem solved for them. Or some prospective customers who pride themselves on being at the cutting edge and wiling to try new things.

Yes, it all takes time, but as we know shortcuts are often shortcuts so the quality can diminish.

Learn more: – A recruiting site (disclosure, we have a partnership with them for our Advanced Participant Recruitment feature)

GV article: How to Find Great Participants for Your User Study

Screener from


Why emotional feedback matters...

Why Emotional Feedback Matters

Consider the last time you made a decision. How rational was it? How emotional were you at the time? Since it is tax season, I’ll share a story of m a recent experience working with my tax-prep software. A few product changes led me to need information from a prior year’s return. You would have thought this was easy, but it turns out it was far more frustrating that you might have imagined. As a user of software as well as someone who designs and build it I found myself cursing at the UX and design teams from my experience, they had not thought this through all that well, and was I livid! If you asked me to type anything at that moment I would have typed everything is ALL CAPS AND PUT AT LEAST FIVE(!!!!!) exclamation points at the end of each sentence.

What kinds of decisions was I making there and why was it so hard to calm down and think rationally? Emotions matter, and if you’re not measuring them you’re missing out on creating the right connection and my emotional connection to this product is not favorable.

This experience by addition of numerous small frustrations lead me to reach a boiling point, or “death by a thousand paper cuts” as the old saying goes. So what’s a UX, design or product team to do? Simply: measure those emotions to see where along the journey are the high and low emotional points and then solve for the sticky, low emotion points. While many customer journey maps include an emotional swim-lane, how many have real emotional data? There are a number of biometric approaches to examining this and getting real data. At Vempathy, we employ facial expression, tone of voice and contextual sentiment to measure this.

There’s a fallacy that we believe that all sound decisions come from a cool head, as noted by neurologist Antonio Damasio in his 1994 book Descartes’ Error.  Except, we don’t really make rational decisions. We make them with our emotions. There have been a variety of studies that show how people with injuries to parts of the limbic system, a part of the brain important in generating emotions, struggle with making decisions.

It’s not like you can just ask your customers outright either. They lie. Not intentionally but they do. Further many companies misinterpret the underlying importance of particular customer experience elements, which ultimately causes a team ship the wrong thing.

To start you need to identify the true emotional points of an experience and only then work to create an emotional connection. Remember that what the customer says directly may not be a true reflection of what they need, so interpreting the emotional signals behind it you can make better informed decisions in your product. The results can be significant: A retail company that focused on creating this type of emotional connection increased its percentage of emotionally connected customers by 5%, which resulted a 50% increase in same-store-sales growth (1).

Bottom line: Emotionally connected customers buy more of the products they feel connected to, and I’m a perfect example. That tax preparation software I referenced? I’m unlikely to renew next year given my recent experience, so if you have any suggestions, send them my way!

Generative vs Descriptive vs Evaluative Research

Have You Mismatched Your User Research?

Let’s consider you are building an aircraft maintenance system for airport workers, is it a good idea to test the prototypes with florists. It could be a very fun test to watch, but you will not learn much from it and aside from some possible GIF-worthy anecdotes you will waste everyone’s time. This is the same when and ask random people to test your product, who may be have diametrically opposite needs from your users. Make sure you match your test subjects with your intended users. 

Here’s another situation: Recall a time that a sibling had symptoms of the flu. If you want to understand how they feel, you don’t use an ear thermometer to measure their body temperature and then make a guess based on their body temperature how they must feel. You straight up ask them how they feel. If you suspect that they might have a lung infection, you don’t ask him to rate how likely they are to have a lung infection. You run some tests.

Using an ear thermometer to understand how someone feels is like running a usability test to understand how much future users would pay for your unreleased product. Asking someone to judge if they have a lung infection is like sending out a survey to understand why people abandon their carts online. Different research questions require different methods to get answers, and the process of picking the right methods for your research can mean the difference between a successfully clearing a lung infection or having to undergo surgery because you didn’t test the right thing. 

Generative Research is employed when you seek to uncover the problem and learn insights from your user base. The purpose is to formulate a problem statement and determine if it is a relevant problem to your business. This is where Shadowing methods and ethnographic approaches work well. Your subjects likely will not state the problem explicitly, you have to synthesize it from your observations.

Descriptive Research is when you have identified the problem but aren’t certain if you have a solution that will effectively solve that problem. So if you have those symptoms of a sore throat, do you have to take antibiotics or would some water and a good mouthwash clear up the soreness? You may have some trial and error here to validate the problem and see if the solutions you are thinking of might old weight. The goal is to have a more detailed understanding of the context of the problem.

Evaluative Research is used when you have a specific hypothesis to test, such as a checkout flow, calendar design, or landing page where you need to evaluate (get it?) how well the design is performing. This type of research you should conduct frequently and it even better if you can integrate into your product design workflow. A subset of this type of research may be causal (not casual!) which would test if A caused B. A simplistic example: Would a bright red text indicate a negative bank balance better than a black text with a negative sign in front of it?

You may think that these research methods are sequential and to some degree, when starting from scratch this may hold true, however all along your product development cycle, all of these methods can help you refine your problem to solve and the way you go about solving it (i.e. your product and your designs).

Ned to run some evaluative studies? Give Vempathy a try, your first project is on us!

Vempathy automatically generates severity scores.

How Severe Are Your Design Choices?

To make your results from Vempathy more actionable, we generate severity scores for each of the tasks that you define in your UX tests. To better understand severity scores we used an industry standard as our basis, the Nielsen Severity Score. Learn more, as it is described on the Nielsen Norman Group website:

Severity ratings can be used to allocate the most resources to fix the most serious problems and can also provide a rough estimate of the need for additional usability efforts. If the severity ratings indicate that several disastrous usability problems remain in an interface, it will probably be inadvisable to release it. But one might decide to go ahead with the release of a system with several usability problems if they are all judged as being cosmetic in nature.

The severity of a usability problem is a combination of three factors:

The frequency with which the problem occurs: Is it common or rare?
The impact of the problem if it occurs: Will it be easy or difficult for the users to overcome?
The persistence of the problem: Is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?
Finally, of course, one needs to assess the market impact of the problem since certain usability problems can have a devastating effect on the popularity of a product, even if they are “objectively” quite easy to overcome. Even though severity has several components, it is common to combine all aspects of severity in a single severity rating as an overall assessment of each usability problem in order to facilitate prioritizing and decision-making.

The following 0 to 4 rating scale can be used to rate the severity of usability problems:

0 = I don’t agree that this is a usability problem at all
1 = Cosmetic problem only: need not be fixed unless extra time is available on project
2 = Minor usability problem: fixing this should be given low priority
3 = Major usability problem: important to fix, so should be given high priority
4 = Usability catastrophe: imperative to fix this before product can be released

When running a user test there are many things to keep in mind.

Five Tips to a Great User Test

When running a user test there are many things to keep in mind to ensure that you walk away with the insights you set out for. We recommend tackling these five areas to design the best user test for your goals:

1. Have a goal in mind, what do you want to learn?
While a “Do you like this: yes or no” might be tempting to lead with, you want to have a clear learning objective with your test. This is not something you might directly ask your participant, either. If you are looking to learn how successful your onboarding flow is, you want to see where in the flow your participants are frustrated, confused or otherwise impeded from continuing. You won’t ask “Were you confused?” but you might ask how they would explain this to a colleague.

2. Identify (and recruit) the right test participants.
Sometimes you have a specific population you need to target, for example, maybe you need to test your product designs with coaches of elite athletes. In such a case it may be wise to source your own participants rather than find them from a user testing panel. Filtering your participants to a target segment, or a “look-alike” segment will get you results that are more authentic and actionable. Vempathy can help you with recruitment when you setup your UX test.

3. Limit the length of the test.
Just as important as having an objective you likely cannot ask a participant for 1 or 2 hours of testing. Limiting your test to five or less tasks can ensure that you have a clear focus and your test participants will complete the tasks you ask of them. IF there are too many, you risk disengaging your participants and this may affect the outcome of your test.

4. Set the stage and give clear instructions for each task, that aren’t leading.
You want to guide participants to navigate the test, but you also don’t want to be overly prescriptive on what to do. You are looking to learn how they use your product, you’re not telling them how to use it. Avoid “click here, then click here” type of task instructions, instead invite participants to accomplish something like “you’re looking to have a physical therapy appointment, find and make an appointment that fits in your schedule.” Instructive but not prescriptive.

5. Test with five to eight participants.
Research has shown that a minimum of five participants will give you sufficient feedback to take action. User experience researcher and professor Bob Virzi concluded that the first four or five participants find 80% of problems in a usability test and additional subjects are less likely to reveal new information. Additionally, he found that the most severe problems are more likely to be detected by the first few participants. The participants need to be of the right profile (see #2) and you won’t need more than seven or eight to start hearing similar points of feedback.

Sketch Plugin for Vempathy

See How Your Users Feel with The Vempathy Plugin for Sketch

In our continued efforts to bring designers feedback from users within their existing workflows, we are excited to introduce the Vempathy plugin for Sketch. Much like with the Vempathy plugin for Adobe XD, Sketch users will now be able to leverage Vempathy’s advanced, end-to-end qualitative research platform.

With the new Vempathy plugin for Sketch, designers can build better digital experiences with rapid customer feedback and analysis powered by artificial intelligence. They will see exactly when test participants express different emotions while interacting with their designs.

“User research is such a crucial part of any product design process and the emotion reports provided by Vempathy’s tools bring a whole new dimension to your testing. This new Sketch plugin makes it easier than ever to incorporate Vempathy’s powerful insights into your existing workflow” said Daniel Duke, Head of Marketing at Sketch.

“We love empowering designers with research tools that can help them do their jobs better and, as Sketch users ourselves, we are thrilled to bring the power of Vempathy to other Sketch users around the world.” -Paul Cheek, CEO, Vempathy.

To get started with Vempathy and Sketch, please visit:

We’ve been there, everyone seems to have an opinion on your design.

Feedback 101: Everyone is a Designer

We’ve been there, everyone seems to have an opinion on your design. “Move the button to the left!” “Make it blue!” and our favorite cliché “Make the logo bigger!” While many designers might want to filter that feedback out, we say: embrace it!

Reactive, Directive, and Critique
There are three types of feedback and some will cause you to roll your eyes. This feedback is often from your colleagues. But what about from your users?

Reactive Feedback
Reactive feedback is where you’d hear an expression such as “That’s awesome!” or “Ugh, that’s was designed by a small blind parakeet.” There’s an emotion here and likely you can identify it not only based on what they express, but also how it is expressed. From your colleagues, this may be a little frustrating, but from your users, this is gold. You need this reactive, expressive feedback from your users and you need to know how to take action based on it.

Directive Feedback
Directive feedback is often in the form of “If I was designing this, I would…” it offers a specific direction on what you as the designer should do. Sometimes these are in the form of a question “Why don’t you move this to the right?” Regardless of how it’s delivered, you can identify these as a form of direction.

Critique is wonderful feedback, when you can get it. Many people, including some designers, are not trained in giving true critique. Critique is when a the objectives of the design are clarified and for each objective, there’s a comment ho how the design has allowed the user to reach the objective. For example: “If your objective is for the user to begin the checkout process, the choice to place the checkout button in grey on the bottom might not be easily discoverable.” You’ll rarely see this type of feedback from your users as they’re not designers nor are they trained in identifying design objectives.

Capturing Reactive Feedback
When you run a test with Vempathy, you’ll capture and quantify the emotions from your users that can help pinpoint your users reactive feedback and turn it into something that’s actionable.