UX Research for Augmented Reality – 5 methods we used to validate PoC

Where is the north?

Remember the last time when you used Google Maps to get walking directions and you heard: “Head North”? How did you feel? We immediately started to look for the North Star ⭐️ How long did it take you to find the right direction? If you ever had troubles with it, believe me – you are not alone.

On the Google I/O 2018 conference the company admitted that starting the route with Google Maps navigation is not a piece of cake. At the same time, they presented an easy solution for the problem – AR hints.https://www.youtube.com/embed/OChF7NNYyq8?feature=oembed&enablejsapi=1&origin=https%3A%2F%2Fwww.tooploox.com

But wait…we came up with this idea first! 🙂 Well, maybe not exactly, but we did start working on the AR navigation way before it was presented by Google.
Meet Hansel – the indoor navigation.

Hansel

Hansel is a startup which was developed internally at Tooploox. At the very beginning, it was just a prototype and before Tooploox decided to invest more money in it, the idea needed to be validated by users.

State Of AR Research

For many years AR has been portrayed as the next big thing. In 2017 Tim Cook said, “I think augmented reality is big and profound”. Apple decided to invest in AR Kit development, which enables developers to create AR-based apps like IKEA Place or Hansel.

Nevertheless, AR still isn’t mainstream – although there are many businesses where AR plays a vital role, a common perception of this term is closer to SF movies than to reality.
Like in Iron Man movie 🙂

Watch more on YouTube

Instead, the reality in 2018 looks more like this.

Don’t get us wrong – we believe that popular solutions help to adopt AR more broadly among everyday people in a more natural way and finally AR will hit the mainstream.

As researchers we had to face the fact that UX research in the AR area is still a niche and there are not many good practices how to do it right (if you want to know more about UX, visit our blog post “What is UX design and how the design process looks like?“). However, we didn’t give up. We rolled our sleeves up, collected a toolbox of well-known methods and tools and tailored them to our needs.

Picking The Space For The Research

Hansel is a product that helps to navigate indoors. For us it was obvious we had to test it in the natural environment instead of the usability lab. For the purpose of our research, we had to pick one space that would meet our expectations like:

easy access to the venue,
easy access to the users of the place.

Eventually, we decided to conduct the research in a shopping mall. Our choice was the newest shopping center in Wrocław to reduce the risk of knowing and moving around the space by heart. Also, the building is quite complicated to navigate. We knew that this shopping center has its own app, so we could use it in the upcoming user tests.

The Analysis Of Existing Navigation Solutions

After doing some initial market research and examining the existing solutions we found out that many shopping centers have their own apps and some of them have a build-in navigation – similar to Google Maps.

We also conducted an online survey among the shopping mall customers. It turned out that none of 81 respondents had ever used such an application, even though the problem of navigating in shopping centers exists. The respondents claimed they had used other methods like asking the shop assistants, looking for some vertical signs or checking the map.

Since respondents didn’t know Hansel nor shopping mall app, we decided to test them both with users and compare two experiences to each other.

Setting Research Goals

We decided to focus our research on emotions and potential of the product. Based on these assumptions, we defined five research goals, which helped us to choose suitable methods.
They were:

examine the emotions that the AR navigation evokes in users,
examine the comfort of using AR technology in a public place,
compare the experience offered by the traditional navigation with experience offered by Hansel,
find the strengths and weaknesses of both navigations,
discover new use cases for indoor navigation.

Problems

If you ever had a chance to test AR mobile apps (MAR), you probably know that the main problem is to create an actual prototype. There is a plethora of prototyping tools for mobile apps but still, almost none which may help you to prototype AR experience.

A few approaches you may try:

paper prototyping,
tools used to create VR content like Google Blocks and Tilt Brush
AR markers

Luckily we had POC app which was fine for testing purposes but still, there were some big challenges to face.

Choosing natural space instead usability lab means we wouldn’t be able to control it – the test might have been interrupted by other people using that space.
Recording how people interact with the application would be problematic. We wouldn’t be able to put the camera behind the respondents’ back and easily capture what they do.

Our first thought was to use gimbal to capture a stable picture but we came to conclusion that it would be very intrusive for others around us and intimidating for our participants.

Finally, we decided on the setup as simple as:

recording the respondents using the smartphone camera held in the hand of one of the researchers,
recording users’ screen by iOS built-in screen recording app.

Methodology

After reviewing different methods that we already knew, the decision was to combine a few of them that would let us get quickest and most precise results possible. In the end, we collected these 5 methods:

Preliminary interviews,
Usability tests – finishing two different routes using both applications,
Evaluation questionnaires for each application,
Reaction cards for Hansel,
In-depth interviews.

Preliminary Interview

The purpose of this part was to introduce the participants to the course of the meeting, make them feel comfortable with the research situation and gather some basic information. We asked them :

what they know about AR,
if they ever used it,
if not – how they imagine it would work,
if they ever used the app of this shopping mall,
how often they visit shopping malls,
what is the most common purpose of these visits.

We also asked them to describe the last time they couldn’t find the way in such a place, to learn more about their habits and identify the pains of this situation. It’s always easier for respondents to recall a real situation than think of examples of their behavior without any context.

An interesting fact: even those respondents who claimed to be tech-savvy weren’t sure how AR works and tended to mix it up with Virtual Reality.

Usablility Tests

The core of our research session was usability testing of both Hansel and shopping mall app. Respondents were asked to reach the destination we set, using each application. There was a different destination for each app so that respondents didn’t learn the route by heart. We changed the order of used apps with each participant.

We also asked the participants to think out loud – comment on what they see, what they like, dislike, don’t understand and how they feel. Likewise, we couldn’t ignore non-verbal signs, which are crucial when studying emotions. All of the tests were video recorded so that we didn’t have to take notes during the testing sessions.

Screenshot from Hansel app from the actual test

Screen from Hansel app from usability test

Screen from shopping mall app from usability test

Evaluation Questionnaire

To capture the fresh feelings and emotions connected with using the apps, each of the tests was followed by an evaluation questionnaire in which we asked the participants to mark on a scale from 1 (negative) to 5 (positive) if:

reaching the destination was easy or difficult,
they had to pay much or little attention to the app,
presentation of the route was clear or unclear,
they felt comfortable or uncomfortable while using the app.

Reaction Cards

Reaction cards is a method introduced by Microsoft and it’s a great tool to use along with usability testing for both new and existing products. Respondents are asked to choose from the list of 118 words they would use to describe a product and then explain their choices. We didn’t want to overwhelm the users by the number of cards but instead wanted to make words more relevant to our case, so we decided to reduce the quantity to 40.

This method can be included in surveys but we preferred to present the words as cut pieces of paper scattered in front of the respondents. The outcomes were really fruitful. The words that participants chose were important but even more insightful were the reasons for picking a specific word. The justification for the selection was a good introduction to further discussion and in-depth interview. Our advice is to always ask people “why”.

The results might be presented as a word cloud:

In-Depth Interview

The aim of the in-depth interview was to summarize the whole meeting, let the respondents add their own observations and gain more in-depth knowledge. First, we asked which of the apps helped them to achieve the goal faster. We didn’t measure the actual time of completing the task, we just wanted to get to know a subjective feeling of the participants. We also wanted to know:

what they liked the most in both apps,
what annoyed them in both apps,
what did they expect from such apps,
if their expectations were met.

The last part was measuring the NPS score – “How likely would it be for you to recommend our product to a friend or colleague?”. We deepened the question by asking who exactly they would recommend it to. Some of the participants pointed themselves as potential users of this AR solution.

Results

AR Requires A Lot Of Users’ Attention

At first glance, results from reaction carts and surveys seemed promising. However, compared to respondent comments during the usability tests we noticed that the level of attention that users had to pay to Hansel was very high and could be a serious obstacle in using the app on a daily basis.

It requires so much attention that I can’t talk to my friend and walk at the same time. Maybe you only need more attention when using it for the first time?
Anna T.

During the survey, the majority of respondents found Hansel’s very absorbing.
While looking at the screen, they stopped seeing what was going on around them. The consequences are as follows:

users do not see which shops (objects) they pass by,
they fear that they may bump into someone or something.

That is a reasonable fear, considering Pokemon Go players who were injured or even killed because of playing while driving or passing the street. There is even a website to report people injured or killed when playing PokemonGo – http://pokemongodeathtracker.com/.

Looking At The Phone Is Something You Need To Take A Break From

When I see that the route leads straight and there are no turns, I can take a rest from looking at the phone.
Anna T.

It seems obvious, but this insight is especially important for our product, at least as long as the smartphone is the most common device to interact with AR. Although devices like Halo Lens or Magic Leap create a huge market hype, they are still only a niche dedicated to special needs. The AR-based navigation requires much more time to look at the screen than a regular navigation. While designing an AR solution you should consider creating a possibility to take a break from the tiring activity.

Using AR May Cause Social Discomfort

Our application requires from the users to hold the phone in a specific way as if they were taking pictures or recording something. This is not a natural behavior for them and makes them feel tense.

Respondents felt strange and worried that:

they enter the comfort zone of the passers-by,
someone may think they are being recorded,
or they are playing a game,
they look strange.

(…) I think that walking with a mobile aimed at people can be embarrassing…. I just want to take a quick glance and then go as if I was using a phone.
Jasiek Ł.

AR-Based Navigation Lets The User Begin The Route Very Easily

I don’t know where I am at all (using the shopping mall application).
Justyna M.

There is no need to know the map (using Hansel).
Justyna M.

Unlike Google Maps or similar navigations, Hansel allows you to start the route without walking around trying to find the right direction. Solving the problem of knowing the map creates a big opportunity in the navigation market.

No Need To Think

You are walking like a donkey.
Anna T.

A big advantage of AR navigation the simplicity. You don’t have to analyse anything, just follow the line.

Conclusion

There are three tips we would like to leave you with when conducting AR research:

Location-based AR apps need to be tested in a natural environment – pay attention to what the respondents are doing.
Emotions are very important in testing AR solutions. Using AR in public spaces may increase the level of users’ uncertainty. While developing your own AR app, keep in mind mitigating users’ discomfort.
Positive feedback may be illusive – the fascination with new technology tends to impact the results. Use other methods to make sure if respondents’ answers are not biased by this factor.

As Google proved – apart from being just fun like Pokemon GO or photo filter – AR technology can solve real problems. The question is how the market will respond to it? What are your thoughts on this?