CoinDesk – a cryptocurrency portal came to the Tooploox product design team with a problem. They planned a redesign of their website and wanted to know more about the audience before getting started. CoinDesk had analytics data but it didn’t tell the whole story.
They wanted to know the motives and behavior patterns behind the data. It was a perfect problem for a user research study. The scope was wide so we narrowed it to cover the basics first and dig deeper later. We had a tight timeline and a global audience — it was clear that we needed to run a remote user research. Otherwise, we might have missed the deadline. So we got to work.
Know who to recruit
We asked CoinDesk about what they already knew about their users. We got data from Google Analytics and marketing segments. Based on data and set goals, we suggested recruiting 7–10 participants, which was enough to allow us to identify major usability issues on 3 different platforms. We also hoped to learn about the needs of users with different levels of expertise. Eventually, we ended up with 7 people due to time constraints. While recruiting, we looked at the following things:
- level of the user’s domain knowledge,
- length of experience with the website,
- platforms they use — 50% of the website traffic came from mobile,
- places they live (we focused on USA, Europe, and India).
Now that we knew who we wanted to invite to the user research study, our next step was creating a screener survey. We created one with Google Forms and CoinDesk team quickly put it on their website. Responses came in fast. We soon had a huge Google Spreadsheet with many results. That’s where “Filter views” came in handy. We set a couple of filters and browsed results efficiently.
It was time to reach out to the people via email. At first, we tried to include people in BCCs, but we hardly got any responses. We discovered that direct messages were most effective when we send then one by one. Sending emails separately made a huge difference. Email clients often give direct messages a higher priority so our invites were seen. We prepared a set of 5 email templates and used them in different stages of scheduling interviews. We saved a lot of time along the way. The stages were as follows:
- Invite to the interview,
- Confirm date & sign NDA,
- Send a link to the meeting,
- Send a link to the survey,
- Send compensation.
Here you can find the template. Feel free to make a copy and use it in your projects!
Take time to recruit the right sample
In the beginning, recruitment went well. We scheduled and conducted 5 interviews but didn’t have any novice users yet. It turned out to be a challenge for us. We thought that about 400 responses in the screener would be enough to pick 2 newbie users. We sent over 40 emails to this group but got very little response. We suspected that new users didn’t feel attached to the brand or comfortable enough to participate in the study. They didn’t have enough reasons to fill out the survey on a website they don’t know well. Even the incentive wasn’t enough. As a result, we decided to put the screener back online to get new leads. With over 900 responses, we managed to set up and run the interviews. Ultimately, the recruitment process took a bit longer than we anticipated but we got what we needed.
In a remote research study, many things can go wrong. The user may not install the required software, the device battery can go dead or the connection can break up. Luckily, we can prepare and avoid at least some of these situations. We wanted to be well-prepared so we ran the pilot study internally with 3 Plooxies. We tested each platform separately: Android, iOS, and desktop.
For conducting tests we used Lookback because:
- It supports all the platforms we needed.
- Participants had user-friendly guides for setting everything up.
- Lookback records the meeting so there’s no room for human error.
- It shows the screen and the face of the participant at the same time.
Lookback informs the user about recording the screen and camera streams. We wanted to be upfront about it anyway and explained to participants why we needed it. It helped us observe non-verbal clues when the user was interacting with the product. We also enabled our own cameras to have a more natural conversation. We noticed that participants often switched windows to see our face to have at least some eye contact with us so it was valuable for both sides.
Why run a pilot?
Thanks to running a thorough pilot study internally we could:
- Predict problems that can come up during the setup process and the interview itself.
- Test the quality of the camera and microphone.
- See what the user experience looks like on different platforms.
- See if there are some Lookback limitations on Android or iOS.
- See if the scenario works well and test it after revisions as well.
- See how much time each part takes.
- Familiarize ourselves with the process. We could feel comfortable in the real study and focus on moderating it.
What we learned after the pilot study
After the pilot study, we noticed a few things:
- The first participant didn’t have headphones and the sound quality was terrible. We quickly discovered that this was something we should remind them about.
- If you print out the scenario and want to look at it, the participant can see that you are looking off the screen. As a consequence, they may think you’re not paying attention to them. So we opened the scenario side by side with the video tab and often looked directly at the camera. We achieved a natural feeling and showed participants that we were engaged.
- In case a participant doesn’t see your face, let them know you’re listening. It’s like during a phone call. Saying “mhm”, “oh”, “I see” or rephrasing some sentences are clues that you’re paying attention.
- iOS didn’t allow us to use the native browser and the user had to stay within the Lookback app. Otherwise, the streaming stopped.
- Chat in Lookback proved to be useful. When the observer had some ideas for the follow-up questions, they could let the moderator know without interrupting.
Before each interview, we sent the participant a reminder about the time of the meeting. We also included a short checklist of things to be prepared — charged battery, headphones, etc.
Lookback on the desktop needs Google Chrome to work. Some people use different browsers so it forced them to switch. This was lame because we had some people who were not familiar with it, didn’t have any bookmarks there or browsing history.
On the other hand, recording interviews was really helpful. Some participants had a strong accent, spoke and acted fast. With the recording, we could go back to it at any time and see what we missed. It made analyzing data much easier than just looking at notes. We also prepared a short video with highlights from the study which we shared with the client. It’s a really valuable tool to enrich insights and show places where users really struggled.
After the interview, we sent a short questionnaire to participants. We asked for extra feedback, associations respondents had with the service and difficulty of the tasks. We asked these questions separately to give users time to pause and think. Later on, we were able to compare the results. In the last step, we sent participants the gift code to an online store as a form of compensation.
Plan remote user research in your next project
For CoinDesk, user research was only one part of the study. We also analyzed data from Google Analytics and a survey we created. The whole process took about 22 days. Preparing and conducting the remote study was a challenge, but a very rewarding one. We strongly believe that user research should be a part of product development. With remote research, there’s no excuse to skip it.
What do we like about remote research?
We ran a remote study and liked it a lot! Why?
- We don’t have headaches from finding the right venue, booking flights, accommodation and inviting participants to the lab, etc.
- Without extra logistics, it saves time and we can focus on things that matter.
- Setting up everything is very easy thanks to services like Lookback or User Testing.
- We have all the meetings recorded by default.
What’s in it for clients?
Remote user research is the right call for many products.
- It’s cheap – we don’t need to rent an expensive venue, cover flights or accommodation.
- It’s fast – depending on the scope, the client can get results in a 2–4 weeks.
- It’s actionable – usually, there are some key insights that could be used right away.
- User research tells more than Google Analytics data. We capture emotions and real experiences of the people using the product.
PS. We also did a redesign of some parts of the CoinDesk website. Take a look at Konrad’s Dribbble shot to see learn more!
Do you want to learn more about your audience? Or maybe you’re planning a redesign and want to make informed decisions? Let’s talk!