Insights | By Howard Tiersky
How to Do User Testing Remotely
FROM Digital’s Director of User Experience Dan Willig details how to get the insights you need during the pandemic—at a lower cost as well
Whenever you want to test an idea, whether it’s a website, a mobile application, a kiosk at a museum, or any new product that you’re developing, user testing is one of the best ways to get feedback.
However, it can be expensive and logistically challenging to get users into your facility, and now during COVID that becomes almost insurmountable.
Remote user testing is a way to still get the insight you need but faster, at lower cost, and with no risk of COVID transmission.
You just need to know some important things about when and how to use this capability to get maximum results and avoid pitfalls.
In the old days, pretty much all user research was done by bringing somebody into our offices or going to another city to do it at a market research site or maybe a meeting room at a hotel where we would have people signed up to come in.
But now, more and more companies are doing this via Zoom or WebEx instead because it just adds a lot of cost when you have to fly to 10 different cities to talk to users there.
It can also be easier to recruit people because they don't have to drive somewhere. They can just do it from their homes.
Although remote user testing is clearly more convenient, there are researchers who feel it isn't quite as good as sitting with people when you can see them up close as they get to those areas that get them really frustrated.
But in fact, there are remote user testing tools and techniques that you can use to effectively find the points where users are experiencing friction.
When doing remote user testing, just as you would when doing any kind of customer research, you can split your subjects into two groups: moderated and unmoderated.
MODERATED TESTS
For moderated groups, you can just get on the phone with someone and share links with them.
Have them walk through things, share their screen and talk about their experience.
It has a lot of the benefits that you get from in-person user testing in the sense that it's conversational.
If someone is doing something, you can ask them what they're thinking at that point and how they felt after they completed a task or weren't able to.
You can also ask them a little bit about what their expectations are, if they are to go to this other area of the experience that you're testing a prototype for but you haven't fully fleshed out yet.
It’s a great way to see the physical cues and the body language when you're able to do screen sharing.
And these sessions are really easy to schedule. You can talk to someone in California and then 10 minutes later you can be on the phone with someone in Michigan.
Also, there's that consistency of the moderator that you don’t get when you have to break up the team and different people will be moderating different sessions.
You get great results still, but without that consistency, it would seem like you're getting a little bit of hearsay.
When you start to look at the data, you wonder what exactly happened in that conversation.
Was the user asked this follow-up question? There may be information that you think could be useful but wasn't captured.
UNMODERATED TESTS
However, there are times when you need to talk to a lot of people and that kind of setup is just not feasible anymore.
There may also be a specific task that you're trying to get specific measurements on.
While you can do that through remote moderated testing, it’s going to take some extra footwork to be done.
That's where unmoderated platforms come in.
Essentially, what you do is create a script for what it is you're trying to test and set all the parameters up.
Of course, everything has to be tied back to your business goals for your product, whether it’s an app or a website on desktop or tablet or mobile.
Introduce the variants that people are going to be having.
For example, your script could start with: “Thanks for taking your time out of your schedule. We're going to be asking you a few questions to help us improve the experience of this website or app.”
And then, give people the exact instructions for the series of tasks that you're going to give them.
It could be: “Go purchase these tires. And when you're just about to enter your credit card data, stop.”
That could be one task.
Oftentimes, you’ll also need to include this reminder in your script: “We'll be capturing your audio and video as you're going through this process. So, please remember to share your thought process and your frame of mind as you're going through the steps.”
The big disadvantage of the unmoderated test is you're not there to probe and ask those questions.
But it's helpful to make sure that you're prompting people of several points throughout the script—tell them to share their thoughts—because people forget.
Maybe things are working very well and they're excited to just finish that they're forgetting to tell you what they're thinking about all of this.
You won't know that until you get the final video.
Most of the available platforms for unmoderated tests have user panels that you can recruit from.
This can be international and you can focus on a particular continent or region if you want.
What you can do is set up demographic controls.
For example, maybe you want to look for men and women in the Northeast who own a car and have owned it for the past two years.
You can set all of that up and then the platform will look out at its user panel and only recruit those people who fit those criteria.
They will be the ones who get the link to go to your test. Once they click the link, they can proceed to take the test.
One of the big advantages of using this method is it makes your analysis much easier.
If you’re recruiting about 60 people for a study, watching the results of all of those videos in order to capture that information is really a lot.
But by setting up a remote unmoderated test, you can literally have 60 videos in a matter of hours—that would probably take a couple of weeks to do moderated either remote or in person.
There are many other benefits.
You can schedule people to take the test at the same time. It doesn’t matter if they overlap.
There’s no need for lunch breaks.
Travel time is not a factor.
If somebody doesn’t show up, you won’t have to sit there for an hour with nothing to do.
All these problems are eliminated.
And you just recruit as many people as you need. With in-person tests, you usually wind up with only 15 subjects.
Another great thing is that you pay panelists less to participate in these remote tests.
It makes sense because you have to pay more to motivate somebody to come to you for an in-person session.
Furthermore, with the unmoderated test, they can do it whenever it’s convenient. They don't even have to plan it.
SOME TOOLS YOU CAN USE
It can be cost effective in that way, although with most of these tools, there is a barrier to entry with membership fees which can be pretty steep.
There are many different products out there to choose from and they vary in strengths and weaknesses.
But for those tools where you have the option to do the unmoderated test, UserTesting.com is the one that first comes to mind because they are sort of the gold standard.
The big downside is that at a certain point, they realized how good they were so they jacked their prices up significantly, which some companies can afford and some just can't.
Validately has an annual membership fee as well, but it's a little bit more palatable and it has a great testing panel.
UserZoom is also great at setting different types of tasks and units of measure that you can use in your research.
While those are all promising, you can try looking more into utility tools where you actually don't have to pay that annual membership and just go month to month.
We’re seeing more and more of these pay-as-you-go models coming up and changing the landscape.
What’s important is that the tool you choose allows you to quickly generate some prototypes and easily get feedback on the general viability of your ideas.
You can read more about customer research in my Wall Street Journal bestselling book, Winning Digital Customers: The Antidote to Irrelevance, which details the different tools for gathering the insights you will need to understand your customer in an actionable way.
Get free access to the first chapter when you click this link https://WinningDigitalCustomers.com.
