Vision+ Mobile App
Usability Testing During Pandemic

Objective

To determine overall design inconsistencies and usability problem areas within the user interface and content areas so that the team can adjust the product roadmap accordingly.

Exercise the application under controlled test conditions with representative users. Data will be used to access whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.

Establish baseline user performance and user-satisfaction levels of the user interface for future usability evaluations.

Challenges

• Scattered design files, even worse half of them were missing due to previous designers resign without any proper documentation.
• No official support (conducted outside the working hours)
• No research budget (no incentives)
• Hard to conduct an in-person testing during the pandemic
• Too many features that need to be validated at once
• People tend to avoid video calls since they overwhelmed already by their work-from-home hustle.

Methodology

Considering all those challenges I decided to go with the moderated remote usability testing using a staging app.

Please note that this design is from previous designers, as is.

Recruited 5 participants. They use their laptop’s camera to show and record the interaction via Zoom app (focus on their hands and phone only).

Metrics

• Completion rate
• Time on task
• Critical errors
• Non-critical errors

Participants Criteria

Currently, we don’t have user persona yet for our products so we use general criteria instead.

• Currently use paid/free Over-the-top media services
• Indonesian citizens
• 13-60 years old
• Use Android/iOS devices on daily basis

NOTES: V+ didn't give any kind of incentives for participants and there is no non-disclosure agreement.

The Draft

Scenario and Tasks

Since it's hard to find respondents without budget and official supports I need to get the most of each testing session. However, I still have to keep it short to avoid overwhelming respondents with tedious tasks that would affect the findings.

The ultimate goals of this study are to uncover as many as possible usability issues in the core features, get the buy-in to reevaluate the design, and establish baseline user performance and user-satisfaction levels of the user interface I tried to make scenarios that could touch those aspects in one session that took roughly 30-45min per respondent.

Senario #1

You just installed the app and want to give it a try.

1. Signup using your preference method
2. Try something you’d like to watch (Movies, Series, TV)

Scenario #2

You finally found what are you looking for but unfortunately you can’t watch it now, you want to mark that movie to be watched later.

1. Add a movie or series to “My List”
2. Find your added movies or series

Scenario #3

You are looking for a movies called “Keluarga Cemara” by using the search feature.

1. Try to find Keluarga Cemara series

Scenario #4

You decided to to buy premium package for 1 month so that you can watch the premium content, now go and finish the payment process until you able to watch the content.

1. Buy premium package, choose the 1 month option
2. Check your premium status

Sceanrio #5

For some reason you need to logout from the app. Now you need to login to enter the app again.

1. Logout from the app
2. Login using the previous account

Pre-test interview

1. What do you do for living? What’s your daily routine looks like? (Weekday & Weekend)
2. Do you currently use paid/free streaming platforms? (music & video) if any, what are they? how do you find them? do you pay for it?
3. What kind of content you love the most? (Movie, Series, TV) please give examples
4. Do you currently subscribe to a Paid TV?
5. How long you spend time watching content? (during your free time & working hours)

Debrief

1. How would you describe your overall experience with the prototype?
2. How would you rate the looks of this prototype? (1-5)
3. What did you like the most from the prototype? Why?
4. What did you like the least? Why? Did you encounter any difficulties as you used it? Which ones?
5. Do you have any suggestions regarding features?
6. What’s your consideration while choosing a streaming platform?

Results

Divided findings per each case and general findings. Here are the snapshot of them.

Findings

Divided findings per each case and general findings and then merge similar issues.

Key Insights

Here are the summarized version that categorized into critical, serious, and minor issues.

Critical Issues

• Only 40% of the participant able to complete the premium package purchasing task.
• There are couple UI pattern violations that caused user get to the wrong page.
• Connect icon placed at the top right screen caused so many frictions to users.
• Basic search functionality such as autocomplete and search results is not working properly.

Serious Issues

• Poor accesibility in couple of screen caused user wasn’t able to read labels in the first attempt.
• Generally, the overall impression regarding the functionality and the design of the prototype (staging app) wasn't good.
• Preference selection that occurs after user signup isn’t working as it should be & users didn’t really engage with the feature.

Minor Issues

• Cluttered & buggy component occurred here and there (affects both aesthetic and functionality of the app).
• Impression and sentiment about the collection of movies/series and movie posters are bad (keep in mind it may be a bit bias since it’s only a staging app).

Problem Severity Matrix

In order to prioritize the issues/insights to stakeholders, I used a problem severity matrix to help classified visually.

Impact

Impact is the ranking of the consequences of the problem by defining the level of impact that the problem has on successful task completion. There are three levels of impact:

• High: prevents the user from completing the task (critical error)
• Moderate: causes user difficulty but the task can be completed (non-critical error)
• Low: minor problems that do not significantly affect the task completion (non-critical error)

Frequency

• High: 60% or more of the participants experience the problem
• Moderate: 21% - 59% of participants experience the problem
• Low: 20% or fewer of the participants experience the problem

Frequency is the percentage of participants who experience the problem when working on a task.

Project Learnings

• Conduct a remote usability testing requires a practice session that should taken into the account, do a quick exercise to warm user up and get familiar with the situation. Always prepare a plan B when internet connection goes crazy e.g educate respondent about how to switch to an alternative and what to do if suddenly disconnected from the call.
• When recording the session make sure the Zoom screen is locked to the respondent side so we won't miss any single moment in the recording.
• Don't make it personal, all your extra miles effort to validate on your own without official supports like this could sometimes be ignored by stakeholders regardless of all the insights were really helpful or not. Sometimes they sacrifice one product to be left flawed to develop new platforms / new features so that they can keep their own KPIs as managers of other divisions. Even when its product has the highest number of potential users that would generate revenue compared to other platforms.