Study Writing & Set-Up Guide

1. Preparation

Set your goals

How to define the goal of your study?

Listen to what’s going on in your business, use them to investigate with a user study and find out why it’s happening – so you can fix it.

First question you must ask yourself is: What are the most common questions you get on support or via email, from your potential customers?

For example:
E-commerce Stores usual issues:

• People are adding things to the cart, but not completing check out.
• Get a lot of support tickets with basic questions about a specific product when that information is already on the site. (What colors does this product come in? Do you have financing options? etc.)
• People can’t find a product on your website, even if you’re selling it.
• There’s a product you believe should be selling more than it is.

Blog or affiliate sites usual issues::

• Do you have information on X? (For example, Do you have any advice on traveling to Queensland?)
• Why should I trust the information on this website?

If you’re marketing a product or service:

• Do people understand what my product is and who it’s for when they land on the homepage?
• Why are people not converting from free trial to paid customers?
• Why does it show people are dropping off at step 3 in my onboarding process?

If you could get an answer to the questions you have, what would be more helpful to know?

Listen to what’s going on in your business. There, you’ll find everything necessary to investigate and find out why it’s happening – so you can fix it.

2. Who is your Target Audience

This helps us create a scenario for the study and choose demographics for the reviewers.You want to find reviewers that are related to the problems you’re trying to solve.

What you should ask yourself: Who are the people that are interested or buying things from my site?

Decide and describe the target audience, lead, or customer that will be the focus of this study.

3. Setting Up the study

The way you write the directions in the tasks has a big impact on the quality and outcome of your studys. As well as what kind of information you want from them.

If you want to know how users are understanding a specific feature, you’d lead them there and ask them to figure it out.

However, if you want to find out how users behave spontaneously and how clear the navigation is, you’d leave it more vague.

Tip: Do not ask direct questions, for example: “do you think the product description is confusing?” Instead, ask them to describe what they understand from your product description in their own words.

study Set-Up Example Information
For the purposes of this walkthrough, let’s say our responses are as follows:

• Website: https://airbnb.com
• Goal: To find out why visitors are going to our experiences page, but not converting.
• Target Audience: A young adult who enjoys traveling and engaging with local cultures, not just the typical tours.
• Questions users have: What’s included?, Is it available on X date?, What are the reviews like?

Writing the scenario

The first thing we want to do is write the scenario based on the information we have. What is the goal of the study and what kind of context do reviewer need to have before they take the study?

First, what is a scenario?

A scenario helps the reviewer get into the frame of mind and understand the goals someone has when arriving on the website. It describes what the user is trying to achieve by providing some context and details to them without giving out too much information.

This puts the user into the mindset of someone who is actually looking to buy X or solve Y problem, rather than the mindset of a person performing a study.

If you want to get unbiased reactions, such as: Do visitors understand what I am selling? In this case, you would use a generic scenario like:

• Follow the tasks speaking your thoughts out loud.
• This is a first impression study.

Creating the Scenario

So let’s open the url: https://airbnb.com and see what steps they’d have to follow to find and purchase an outdoor experience in a specific city for less than $80.

Now, we want to think of the person who is looking for and buying this and what kind of situation they could be in.

Do’s:

Good scenarios provide a bit of context so that users can behave as if they actually need to perform the task. We’ve done this by providing a realistic backstory they can think about when taking the study.
provide specific details the user needs to know. These can be prices, items, brands, dates, etc. We’ve done this by providing dates (February 2-3), location (Medellin Colombia), and the budget (less than $80).

Don’ts:

Don’t put words from your interface or website into the tasks. For the example above: notice I’ve used “tour” instead of “experiences”. You don’t want to lead them to the answer by telling them the exact word they need to click on, you want them to figure it out by themselves

Scenarios are not directions. Aim to provide details and context without telling the user what to do. Don’t write: Go to this website and click on “experiences”. Then in the dropdown select the city, Medellin, and input the dates August 2-3. You won’t get good feedback doing that as you’re telling them the answer.

Writing the Tasks

Note: The maximum amount of tasks you’ll be able to create is determined by the “study length” options which I’ll outline in the next section.

You want to set clear goals that define the success of these tasks so the user knows when the task is completed. If you ask the user to sign up for your service, make a note that this task isn’t completed until the user reaches X or does Y.

This could be a confirmation page, when asked for payment details, or a specific step.

Example study:

Task 1: Give your immediate impressions of the site without scrolling—What is your first impression? Do you trust that you could get a good quality product or service here? Why or why not?

This task helps us understand their first reaction. Do they trust us and what is their gut reaction on the quality of our tours before they even begin their experience?

Task 2: Find a tour for you and a friend in Medellin on Aug 2-3 for under $80 per person, compare the results available and choose one, explaining why you chose it.

This task helps us understand if they able to navigate the site easily, find options that meet their requirements, and have enough information to make a choice?

Task 3: Can you find out what is included in the tour? Are there any extra fees?

This task helps us understand if they’re able to find more detailed information about the tour and if it’s accurate.

Task 4: If this wasn’t a study and you were really booking a tour, is there anything you would do on or off this site before purchasing? Do it and walk us through your experience.

This helps us understand their natural flow. Do they go to a competitor’s website? Do they look at your reviews on google? What do they do? Where do they go? This helps you understand the buying and decision-making process.

Task 5: Add to cart and check-out until you reach payment information then stop.

This helps you see the add to cart process, anything that’s confusing, or if a user encounters bugs along the way.

Choosing the study Length & Number of reviewers

Your study length can vary depending on how many tasks you write and on the number of users you need to study your website.

First, think about your goal of the study and the tasks required for that.

Then, think about how long it might take someone to complete those tasks AND talk through them. Even if a task takes a minute to complete, it will take around 2-3 minutes for reviewers to complete the task and also talk you through it.

If you have 3 tasks, but each task would take a user several minutes to complete (like a software onboarding process), you’ll need to purchase a larger study (even tho it fits into a smaller size). This is for the extra time each task would take to complete and talk through their experience.

To keep things so you only pay for what you need, we divided the study length into 4 tiers.

First impression: Learn what users think about your site at first glance.
Tasks Allowed: 1
Time Cap: < 5 minutes

Short & Sweet Quick answers, in short simple studies.
Tasks Allowed: 3
Time Cap: 5-10 minutes

Basic Flow: See how users move through a funnel. <--- Most studies fit here
Tasks Allowed: 6
Time Cap: 10-15 minutes

Deep Dive: See how users deep dive into the experience.
Tasks Allowed: 12
Time Cap: 20 minutes

How Many Reviewers

We recommend 5 reviewers for the best efficiency and ROI. With 5 reviewers you get 85% of issues in a flow. It’s better to do more studies with 5 reviewers and iterate 4 times, than do one large study with 20 reviewers. – Law of diminishing returns.

If you’re on a budget, we recommend 3 reviewers. This is enough to get you feedback and make key changes.

As soon as you select any of the study size options, you’ll see the total cost of your study getting calculated automatically on the top right-hand side of the page.

4. Analyze results & repeat

Once you watched the reviewers try and do those tasks, you will have a pretty good idea of the problem and why people are having issues. You will also learn how to make your tasks better! As long as you get the basics right, you’ll get good feedback and learn how you could improve your task instructions for future studies!

The goal when watching your videos is to narrow down what worked and what didn’t and decide what action steps need to be taken.

This allows you to create a hypothesis for fixing it.

Once implemented you could run another user study to see if it performs better or worse than the previous iteration.

Any questions? Hit us up in support chat! 🤓