Conducting a Usability Test for TechComm Students

Welcome to the Usability and UX Online Training module for Technical Communication Students. In this module, you will learn the basic concepts of usability testing as well as how to conduct your own usability test. Towards the end you will learn effective strategies on how to analyze your findings as well as how to use them to form effective recommendations for your client.

Section 1: Usability Basics

What is Usability?

Usability is defined as how something is used by specific individuals to accomplish goals effectively, efficiently and satisfactorily under a given context of use.

When we talk about a product that is being used by specific individuals, we’re talking about the target audience that will be using the product to accomplish tasks that they would need to complete to achieve a goal.

You’re going to be looking at three factors when it comes to usability which are effectiveness, efficiency, and satisfaction. When a product is effective and efficient, it means that the product can help a user perform a task in a manner that respects the users time. In other words, the product should be able to help a user perform a task in a way that’s better than they've been doing it.

Satisfaction is the way in which the user perceives the effectiveness and efficiency of the product they’re using. The user’s satisfaction doesn’t just fall on how effective and efficiently the product accomplishes a task. How the product looks and feels can drastically affect the user experience which in a lot of cases can cause the user to overlook any small problems with the product.

What is Usability Testing?

One important point to get out of the way regarding usability testing is that the focus is put entirely on the user rather than the product. How a user feels about the interaction with a product provides valuable insight into how to improve the user experience of the product. And that is what usability testing is all about; it’s about observing a user that is using a product to perform a task that is meaningful to them.

The growth of technology allows for testers to be able to conduct usability testing in a variety of different contexts and a variety of strategies and software which will be discussed throughout this module.

Formative and Summative Studies

Formative Studies

In usability testing there are two different types of studies that can be conducted, and they are called formative and summative studies.

Formative studies, or small studies, occur during a products development. A usability team will conduct a formative study to find problems that need to be addressed before the end of the products development. Formative studies are repeated in increments throughout until the end of the products development.  


Summative Studies

Summative studies, or large, studies are conducted after the product has completed its development cycle and is conducted to gather quantitative or to see whether the final product meets expectations. Summative studies are typically longer studies and require larger sample sizes for to attain accurate results.

Some summative studies need to be conducted during a products development cycle. An example of this are products that require high usability due to potential risks to the user, or if additional data is needed to convince management of product changes.

Testing Environments- Lab Testing

Using an on-site lab has many advantages for conducting usability testing. Testing in a lab allows you to use a controlled environment for testing that is always readily available with the necessary equipment. Labs can be maintained and upgraded if needed. The essentials for a lab are:

  • A desk
  • A chair
  • A computer or laptop
  • A microphone
  • A camera

Testing Environments- Field Testing

During field testing you will be going to the users rather than the user coming to you. Conducting field testing can be advantageous because it will

Pros of field testing:

  • Allows you to gather information on the environmental context of the products use such as types of hardware, lighting conditions, connectivity, and any potential distractions.
  • Insight into the impact of distractions
  • See what users use to support accomplishing tasks


Cons of field testing

  • You are not able to control the test environment.
  • There is a high chance that the participant can get distracted by workplace or home obligations.
  • The participant could be uncomfortable doing a think aloud protocol in an environment where others can hear.
  • Corporate guidelines may not allow for product testing at their site.

Testing Environments- Remote Moderated Testing

Moderated testing means conducting a usability test with a moderator present for the test session but not in the same physical space as the participant.

Pros for a moderated test:

  • Its affordable
  • Brings participants to the test
  • Possibility for a diverse sample


Negatives for a moderated test:

  • You are unable to see the participant and the product at the same time
  • Moderating is more complex.
  • More work to establish rapport between the participant and the product.

Moderated testing can be conducted using online tools such as GoToMeeting and WebEX. These tools allow you to communicate with the participant using VoIP, and you can ask them to think out loud as they complete their tasks.

Testing Environments- Remote Unmoderated Testing

Unmoderated testing is used to test participants no matter where they are, often through means of a web-based application and without a moderator present to oversee the session.

Advantages of unmoderated testing include:

  • Competitive analysis
  • Benchmarking tests to gain performance metrics
  • Analyzing comments made by your participants


Disadvantages of unmoderated testing are:

  • You are not able to see the participants during the test session.
  • You are not able to speak with the participant.
  • It’s expensive

When would you use a Formative Study?

  • At the end of the products development.
  • Iteratively, during the products development.
  • When you need to show metrics to stakeholders.
  • Only when consumers complain about errors in the product.

What is a benefit of field testing?

  • You save money because you don't use a lab.
  • You get data on the environmental context in which the product will be used.
  • All companies allow for field testing at their locations.
  • It's easier to do a field test compared to all the other methods.

What is Usability?

  • The study of how something works.
  • When a product is used by a user to effectively and efficiently use a product that produces satisfactory results.
  • A means of measuring how broken a product is.
  • When a product is used by a user that delivers results no matter how they feel about it.

What three factors do we look at when we measure usability?

  • Effectiveness, Efficiency, Satisfaction
  • Quality, Quantity, Design
  • Effectiveness, Quantity, Durability
  • It doesn't matter. A product has good usability as long as it works.

What test method allows you to test users no matter where they are? Often done using web applications.

  • A Heuristic Evaluation
  • A Field Test
  • An Unmoderated Test

What are disadvantages of doing an umoderated usability test?

  • You cannot see your participant
  • You cannot speak to your participants
  • It's expensive
  • All of these are true

Section 2: Planning the Usability Study

Conduct a Heuristic Evaluation

Before planning the usability test, you will need to conduct a heuristic evaluation. A heuristic is a set of guidelines or “rules of thumb” determined by experts to evaluate the user interface of products in development. Heuristic evaluations are necessary, because they allow usability experts, or experts in other fields, to identify infractions in the user-interface, which will help them determine what to look for during a usability study.  

Begin by choosing three to five evaluators that will perform the heuristic evaluating.

Each evaluator will review the product at least twice. The first evaluation is to become familiar with the product and to generate a first impression. The second is to evaluate the product against the set of heuristics.

Affinity Diagram

Once the evaluators have completed the heuristic evaluation, analyze the results for any overlap. A good way to do this is to assemble an affinity diagram. Affinity diagrams allow each evaluator to record each finding on a card, line them up under a criterion, and then count how many correlations there are under each category. This is a good way for the usability team to find common problems that need to be addressed. 

Schedule Your First Planning Meeting

Your first planning meeting is going to be the most important meeting you’re going to have with your usability team. Here you will plan the entire usability study. During the first meeting you will need to:

  • Identify your test goals
  • Figure out how you’re going to test the product
  • Identify user subgroups
  • Determine the participant incentive
  • Draft the screeners
  • Create scenarios
  • Determine the qualitative or quantitative feedback methods
  • Set dates testing, test material drafts, final test materials, test set up, posttest deliverables

Your first planning meeting is going to be the most important meeting you’re going to have with your usability team. Here you will plan the entire usability study. During the first meeting you will need to:

  • Identify your test goals
  • Figure out how you’re going to test the product
  • Identify user subgroups
  • Determine the participant incentive
  • Draft the screeners
  • Create scenarios
  • Determine the qualitative or quantitative feedback methods
  • Set dates testing, test material drafts, final test materials, test set up, posttest deliverables

Determine the Test Goals

Establishing you test goals is one of the more critical first steps to planning out a usability study. You test goals are what focuses on what you want to learn about your user’s experience to help you develop a product and should be the first thing you plan in the first group meeting. To help you identify the test goals, you can communicate with your company’s customer support, or use the results form a heuristic to find out what part of the product a user struggle most with. You can also use the 5E’s to determine what the test goals are:

  • Efficient: Can users accomplish a task quickly and with no help?
  • Effective: Can the user accomplish a task using the product?
  • Engaging: Are users having a positive experience with the product?
  • Error tolerant: Do users make frequent errors? Are they able to effectively recover from these errors?
  • Easy to learn: Are users able to learn the product quickly?

Build a User Persona

Usability is about the user, not the product, which is why it’s important to understand who the users of your product are going to be and what they want to accomplish with it, and this is where user personas come into play. A user persona is a fictional representation of a product user based on what is already known about the user.

Personas should have the following characteristics:

  • Name and picture
  • Demographic information
  • Job title
  • Goals
  • Environment
  • Technical expertise
  • Quote that describes important factors to a persona

It’s also a good idea to include smaller details in a persona, such as favorite hobbies in order to flesh them out. Doing this will help you think of your persona as a real person rather than just a piece of demographic information.


Things to Consider When Building a Persona

Things to remember when building a persona:

  • Consider your time and budget when building your personas. The number of personas you build depends entirely on what you want to know and the resources you have.
  • Keep your personas visible. It’s important to have the personas readily available for the team to view as they work. It’s a good idea to display your personas in areas where you team meets the most.
  • Don’t get carried away when building personas. While it’s good to include details to flesh out a persona, they should only include information that is relevant to the product in development.

Agree on User Subgroups

User subgroups are user groups that have specific characteristics that are relevant to the product. For example: If you are testing a website you would want a specific subgroup to be familiar with the type of product you are making, uses the internet x number of hours a week, has computer and software skills. Another subgroup would be more of a novice user with little knowledge of the product, doesn’t use the internet much, and lacks computer skills. If you have a user persona this process is easier because you have a foundation from which to start from but there can be several subgroups under a single persona.

Determining How to Test the Product

When determining how to test a product you will first need to determine where the product will be in its development cycle. You will normally be testing a product iteratively throughout development, this is useful because you will be able to present your findings to stakeholders to produce user centered results. In some cases, however the stakeholder may wish to hold off on usability testing to conduct a summative study.

Once you have decided on how to test, you will need to determine where. It’s most ideal to use a lab if you have one, but a conference room or a similar space is also appropriate.

You may decide that you need to conduct a test in the context of the products use, so a field test would be an appropriate choice. In the field, you will be able to get data on the products environment, such as lighting, or whether it’s a wet or dry environment.

Testing remotely using software is a good option when you want to reach out to the user. This will allow you to conduct the test with the user in their environment.


Product Testing Methods

Next you will need to choose the method of the usability test. This can be done in four ways:

  • Testing with scenarios- Testing with scenarios will allow you to present tasks relevant to the user that will allow you to gather feedback based on their experience.
  • Benchmarking the product- Benchmarking is done to establish a set of metrics or specifications for a product. This sort of usability test is typically summative and conducted toward the end of the product development cycle.
  • Design comparison- A user will choose from a set number of designs to see which one they prefer.
  • Competitive evaluation-  Competitive evaluations will allow you to gather data on competing products in to learn about what works or doesn’t work. You can then apply that data to improve upon your product to gain an advantage.

It’s worth noting that the method you choose will affect he scenarios as well as what you will be able to accomplish during each test session.

Create User Scenarios

When creating your scenarios, you need to create tasks that your personas would be realistically trying to accomplish with your product in their context. Your scenario should be a short paragraph that uses your personas language telling them what their task is but not how to accomplish that task. 

Use the first scenario as means of means of capturing your participants first impression of the product. You can only get one first impression, so any feedback here is going to be valuable. Have the user examine the web page or UI of the product and ask them questions such as:

  • "Whose site do you think this is?"
  • "What would be the first action you take?"
  • "What is your first impression of the product?"

The rest of your scenarios should be geared to specific tasks that you want the participant to do that are relevant to them. You will need to set priorities to which scenarios you choose, because you may not have time to have the participant perform all the tasks that you have in your list.

You will need to determine how you are going to let the participant know when they have completed the scenario. You can usually accomplish this by including a request in the scenario instructing the participant to let the test moderator know when they have completed a task.

Determine the Participant Incentive

People are busy, and time is a precious commodity, so it’s important to come up with a incentive to reward people for the time that they give you. Rewarding people with monetary compensation is the most typical incentive. The amount that you give depends on the location of the study as well as your budget. A $50 to $60 incentive is enough for most people; however, specialists will require a larger incentive that will most likely be over $100. Rewarding participants with gift cards is also a good way to go. You want to make to pick gift cards that participants can use anywhere, so VISA gift cards could be the best way to go.

Your budget may not allow for an incentive which means you will have to change your recruiting strategy. Friends and family will generally help without the need for an incentive though you will try to recruit as close to your real users as you can.

Draft the Screener

You will need to put together a screener to help you recruit participants for the study. The screener should include questions that ask the participant:

  • Demographic Information
  • Level of experience
  • What similar products they own or used
  • Technical knowledge
  • Availability

Write the Test Plan

In order to help you keep track of you plans for the usability study, it’s important to document all the decisions that you’ve made. The test plan allows you to put all the decisions you make into writing to keep you from forgetting about them and to make any necessary revisions to the plan. Your test plan should include the following:

Your test plan should include:

  • Product and test goals
  • Status of the products development
  • Issues with the current product version
  • New product features
  • User profiles

Why are Heuristic Evaluations Used?

  • To see if a usability test is actually needed.
  • To come up with new design ideas.
  • To evaluate the product against UX best practices.
  • To determine how many participants you will need for the study.

What should be done first when planning a usability study?

  • Establish the test goal
  • Not much planning is needed.
  • Begin recruiting participants
  • Draft the test script

What product testing method should use use to determine specification data?

  • Competitive Evaluaion
  • Design Comparison
  • Benchmark
  • Scenarios

When building a persona, its okay to use random details as long as their a realistic.

  • True
  • False

What is a affinity diagram used for?

  • To find correlations in findings from a heuristic evaluation.
  • Its used to design a new user interface layout.
  • It's used to determine what test roles team members should take based on their skills.
  • It quantifies the errors found during the heuristic evaluation.

What is the screener used for?

  • Used to help recruit qualifying participants for a study.
  • Is filled out by the participant in order to consent to recordings.
  • Used to help build personas.
  • The screener views the participants screen during the study.

Section 3: Preparing the Usability Study

Recruiting Participants

There are many sources for you to recruit participants yourself. You can draw upon the customers that your company already has using their website or database. Doing so will make it easier to find customers that meet your qualifications for the study. Using your network of family and friends is also a good idea to find participants.  Ideally, you will want to recruit slightly more participants than what you need so that you have extra’s in the event that participants don’t show up to the test session.

Recruiting from Websites

Using websites such as Craigslist can help you search your local town or city for participants. Doing this means that you are going to have to screen more carefully so that you can weed out any potential “professional participants”, those who frequently respond to test queries for the incentive.

Recruiting From an Agency

If you are going to recruit from an agency be prepared to carefully go over your screener with an agent, especially if this is your first time using an agency. Your screener should be clearly defined when it comes to who will be your participants for the study as well as what methods you will be using. It’s also important to establish whether it will be you or the agency that will be providing the participant incentive.

If you are going to recruit from an agency be prepared to carefully go over your screener with an agent, especially if this is your first time using an agency. Your screener should be clearly defined when it comes to who will be your participants for the study as well as what methods you will be using. It’s also important to establish whether it will be you or the agency that will be providing the participant incentive.

Assign the Team Roles

Now it’s time to determine who is going to do what during the test session. There are three roles available for the usability team to take: the moderator, and the logger/note takers, and observers will watch the study from another room.

The test moderator interacts with the participant throughout the study and ensures that the session is progressing. The test moderator role is one of the more challenging roles due to him having to be consistently personable across all test sessions.

The logger/note taker will be using logging software such as Morae to record any observations made during the test session. The logger should be the team’s fastest typist so as to keep up with the test session. It’s also helpful that the logger is already familiar with the product that is being tested to ensure that the notes are accurate. More than one person can serve in this role.

Observers view the study from another room. This group will usually consist of corporate management and people from the web development team. 

Preparing the Test Script


A test moderator is going to need a test script for every session. The test script is what is going to help the moderator interact with the participants in the same way for every session.

The moderators test script should be written to do the following:

  1. Greet the participant- Thank the participant for taking the time to participate in the study.
  2. State the purpose of the study- State the goal of the study and that you wish to hear the participants thoughts on the product, both positive and negative. Reassure them that you are testing the product and not them.
  3. Provide any forms required- Provide any permission forms to the participant if you are going to be using the camera to record them to be used later.
  4. Describe the test room- Describe the room that they are in and along with any equipment that is in the room. Be sure to explain what you will be using the equipment for.
  5. Explain the test session- Explain what is going to happen from the beginning of the test session to the end. Explain that you will be observing the test session and that you will standing beside them throughout the test.
  6. Describe think aloud- Explain to them how think aloud works and why you are using this method.
  7. Ask the participant if they have any questions- Ask the participant if they have any questions before the test begins. Be sure to let them know that once the test begins you will not be able to answer questions until the end of the test session.

Preparing the Video Consent Form

You will need to create a video consent form to record your participant. The consent form is served to the participant before the beginning of the test session. Your consent form should include the purpose of the study and if any of the recorded material is going to be shared with anyone. You will also need to work out how the participant is going to be identified whether its by first name or a number. Most importantly the form should make it clear that test has the power to stop the test at any point during the test session.

Creating the Post-test Questionnaire

The purpose of this questionnaire is for the participant to be able to rate the overall experience with the product. When creating the questionnaire, you are free to ask both closed and open-ended questions.

The moderator observes the test on the side and takes notes if necessary.

  • True
  • False

Why do you create a post-test questionnaire?

  • To get quantifiable data
  • So that the participant can rate the overall experience
  • To see how likely the participant will recommend the product.
  • All of the above

What should not be included in the moderator test script?

  • Description of the think aloud protocol
  • Description of the test layout
  • Responses to participant questions asked during the test session.
  • The scenarios

You are always required to provide a consent form when recording a participant.

  • True
  • False

What information will you need to use a recruiting agency? (select more than one)

  • Test dates
  • Participant qualifications
  • Test methods
  • Your budget

What team role is responsible for taking notes during the study?

  • The moderator
  • The logger
  • The observer

Section 4: Conducting the Usability Study

Greet and Brief the Participant

  1. Thank the participant for taking the time to participate in the study.
  2. State the goal of the study and that you wish to hear the participants thoughts on the product, both positive and negative. Reassure them that you are testing the product and not them.
  3. Provide any permission forms to the participant if you are going to be using the camera to record them to be used later.
  4. Describe the room that they are in and along with any equipment that is in the room. Be sure to explain what you will be using the equipment for.
  5. Explain what is going to happen from the beginning of the test session to the end. Explain that you will be observing the test session and that you will standing beside them throughout the test.
  6. Explain that you will leaving the participant to work alone on the scenarios. Make sure that they also understand that you will be asking them questions during the study and that you won’t be able to answer any of theirs until the end of the test session.
  7. Make sure that they understand that you are testing the product and not the participant.

Describe Think Aloud to the Participant

While it is a strange request, ask you participant to think out loud as they complete their scenarios. Explain to them that they should be verbally communicating what comes to their mind no matter if its positive or negative feedback or if they are stuck on unsure about something. Explain that this process allows you to better gather data about their experience.

Begin the Test Session

Read from the test script throughout the entire test session. Once you have greeted, briefed the participant, and asked them to think aloud, serve the first scenario. Be sure to read the scenario out loud to the participant and let them know to notify you when they have completed the scenario. Keep repeating this process until the participant has completed all the scenarios. 

Ask Good Questions

Ask the participant good questions. Typically, good questions begin with “How” and “Why”.

Here are some example questions:

  • “What are you trying to do know?”
  • “What would you do if you were the only one here?”
  • “Why do you think that happened?”

You may be tempted to answer questions that you participants ask you… don’t. Instead you should probe them with other questions such as “Did you expect that to happen?”

Intervening in the Study

Sometimes, despite all your preparation, things can go wrong, the computer crashes, the participant struggles, or distractions occur.

Should the system crash or become unresponsive, intervene immediately. Briefing the participant beforehand about the possibility of a crash will reduce the chances of them trying to troubleshoot on their own.

If the participant begins to become frustrated while trying to complete one of their scenarios resist the urge to help. You will not want to wait for too long; however, and it’s going to be up to you to decide when you will need to intervene.

Distractions can happen when the participant wanders from their scenarios or becomes hopelessly lost in the product. This can waste a lot of session time and can keep you from gaining the results you want from the test. In this case intervene if they do not find their way back.

The Post-test Questionnaire

Serve the posttest questionnaire to the participant once they have completed all the test scenarios. Make sure that the participant communicated their answers to the questionnaire orally so that observers can take notes of their answers.

If you are using Morae for the study, use the SUS questionnaire so that you may display it on the participants screen and you may see their responses and hear them vocalizing their answers.

Thanking the Participant and Providing the Incentive

Once the participant has completed the post-test questionnaire, thank the participant for taking part in the usability study.

Answer any further questions that the participant may have about the study.

Give them the incentive and see them out of the test area.

What is an example of a "good" question?

  • "Do you need any help?"
  • "What did you expect to happen after selecting that link?"
  • "Are you feeling lost?"
  • "Do you feel frustrated?"

Which of the following situations would require intervention by the moderator? (select more than one)

  • The computer locks up.
  • The participant seems confused.
  • The participant wanders from the scenario.
  • The participant commits an error.

What should be part of the participant briefing? (select more than one)

  • Describing the layout of the study from beginning to end.
  • Giving them a copy of the screener.
  • Describing think aloud.
  • Letting them know you are open to help them if they need it.

You should resist the urge to help the participant unless they are truly lost in the product.

  • True
  • False

You should read the scenario out loud to your participant when they are first introduced to it.

  • True
  • False