7c. Evaluation and iteration

Evaluation is all about gathering feedback on your prototype to quickly identify aspects of the concept that may need to be changed.

A few things to keep in mind during evaluation:

• Sometimes you will discover that a concept is not going to work - and that's okay! If that happens you can take a step back and create a new prototype for one of your other most promising solutions.
• You may need to conduct several rounds of prototyping and evaluation before you land on a successful concept.

To get feedback on a prototype, begin by giving the user an introduction to when and where they might encounter this product or service in the real world. For example you might say, “Imagine that you are riding the bus and using this new app on your phone."

When you introduce users to a prototype that has an interactive element, you may need to take a “magician” approach - meaning you will simulate interactivity by moving, removing, or adding elements to the prototype based on how the user interacts with it during your feedback session.

To see an example of how the magician approach works in a user feedback session, take a look at this video: 

Video from lukenwarm on YouTube

The DOORS principles that you used while conducting user research are also useful for gathering feedback on your prototypes.

As a reminder, the DOORS principles are:

• D: Digging deeper

• O: Open-ended questions

• O: Observation

• R: Rapport

• S: Stopping bias

As in your earlier user research, it can be beneficial to prepare a brief discussion guide with the questions you plan to cover during each session.

The following prompts can be good discussion starters when gathering feedback:

• “What is your initial impression of this concept?”

• “What is your reaction to the shape/size/functionality of this concept?”

• “Could you see yourself using this concept in your own life? Why or why not?”

• “What is most appealing about this concept? Why?”

• “What is least appealing about this concept? Why?”

• “How would you describe this concept to a friend?”

Be sure to document your findings through notes, photos, and audio or video if possible.

When you’re sitting across from a user and asking for their feedback on a concept you’ve developed, it can be difficult to avoid showing your emotions in response to their reactions.

But as a facilitator, it’s important you avoid emotionally responding to the user’s feedback (positively or negatively) and focus instead on understanding their point of view.

Keep in mind that users may be intimidated or nervous about giving you negative feedback on your concept. Because of this, when time and resources allow, it can be helpful to have a neutral person who wasn’t involved in creating the concept facilitate the feedback session.

The goal of receiving feedback on your prototype is to understand what you can change to improve the design and ultimately create a better solution.

Typically you will not to make changes to your concept in response to every single piece of feedback you receive from each user. Just as you looked for patterns and themes in your user research data during the earlier analysis phase, you should look for patterns and themes in the feedback you’ve received from users in response to your prototype.

If several users offer criticism of a particular aspect of your design, or make a request for a new feature, that can be a good indicator that you may want to address that in your design revisions. If one or two users raise an issue but feel very strongly about it, that can also be an indicator of something that warrants follow-up.

You may discover through prototyping and evaluation that an idea just isn’t going to work. Maybe through the process of building a prototype you realize that there’s something about the form or size of the object that won’t be feasible. Or maybe you gather feedback from users that suggests they don’t think the concept is valuable.

It may not feel like it, but this is actually a step in the right direction. “Failing” like this, particularly early on, allows you to better understand what your solution should not be like.

Early failure gives you an opportunity to correct course - either by making major changes to your concept or by selecting another concept from your ideation stage to explore - before you’ve invested too much time and effort in building your concept further.

A defining feature of the design thinking process is iteration. After you’ve developed an initial prototype for one idea and gathered user feedback, iteration typically involves:

• Making changes to your concept
• Creating a new prototype
• Doing another round of evaluation
• Repeating the above steps

The goal is to keep improving your concept so that it better meets people's needs. Keep in mind that you may go through several rounds of prototyping and evaluation to come up with a great design.

Sometimes iteration may involve going back to earlier stages of the design thinking process. For example, you may realize that a new question has emerged about what people actually need or do, and you will need to return to the empathize stage to investigate that topic.

An example of a prototype becoming more refined

As you iterate and refine your concepts, your prototypes will likely become more refined as well. You may add additional details or features, or increase the fidelity of the prototype so that it looks, feels, and functions more realistically.

Just because an idea works well as a prototype it isn’t guaranteed to be a success. Once you launch a product, it’s important to continue gathering feedback and making changes as needed. Failing to adapt to local contexts, or to plan for the entire lifecycle of the product (e.g., what happens when it breaks?) can prevent seemingly great products from being used.

An example of a product that seemed like a great idea at first, but largely failed to be used once it was launched at a larger scale, is the PlayPump, a merry-go-round designed to pump water.

In a project like this one, prototyping the concept early on, to be used in context and with users, may have surfaced some of the issues with this concept.

Learn about the failure of PlayPump here.

With your problem-specific learning group, discuss:

• What are the pros and cons of failing early and often?
• Do you consider yourself comfortable with failure? Why or why not?