One of the most powerful is the double diamond from The British Design Council, which in turn uses divergent-convergent thinking. A clearly-written plan will also help you explain the goals of the test to your team, and help you achieve buy-in, if you need that. Once you’ve launched your study and gotten your results back, it’s time to get to work on analysis. Thank you!Check out your inbox to confirm your invite. Hope that helps! The process of turning a mass of qualitative data, transcripts, and observations into an actionable report on usability issues can seem overwhelming at first—but it's simply a matter of organizing your findings and looking for patterns and recurring issues in the data. This could be roughly compared to the business value in agile methods. What is the cost/benefit of running an experiment to find out? It’s one thing to ask the opinion of users—they’re usually willing to share—but it’s another thing entirely to ask questions that get to the heart of their experience. Was your process more issue oriented because you are assuming that positive feedback wouldn't actually help to make your product better ? Quantitative usability testing tends to generate large data sets, which can lead to problems like analysis paralysis and fatigue. Plan the usability testing and analysis phase. Users with more experience tend to perform better on tasks, have higher perceptions of usability, and have higher Net Promoter Scores. The higher the ROI, the better. Some epic stuff here. 13 Nov 2019. To reduce the risk of making bad design decisions, we need: a) several solution alternatives to choose from, and b) an effective selection process. Finding users: You can recruit users right off your website using a pop-up invite, email users from an existing customer list, or use a panel agency that finds users who meet your requir… Just translate this into a quantifiable value, like the Fibonacci sequence (1, 2, 3, 5, 8, etc). Usability testing can save you a ton of time and money by revealing issues while they are still easy to fix. By Tarik Dzekman. Website analysis is the practice of testing and analyzing a website's performance in relation to SEO, speed, competition, and traffic. At the end of the exercise, the solutions in quadrant 1 are the ones with best ROI (more effective and less complex) signifying top priority. You can start a usability evaluation at the prototyping stage and repeat it with each update to ensure they don’t introduce any new issues. Let’s start by borrowing some ideas from the creative process. What are the resources required to develop this solution? Benchmarking is a great way to measure results against a goal. Each project will require a different approach. Identify the users to test: You probably have some idea about who the users or customers are that come to your website. Fostering collaboration through “quick and dirty” visual analysis at the likely cost of accuracy is a potential trade-off. Starting with your research questions, the first step is to collect the data generated by the usability test. Read on for our best practices. The 6 steps for running unmoderated usability testing are: define study goals, select testing software, write task descriptions, pilot the test, recruit participants, and analyze the results. If you are doing this as a team, planning poker fits perfectly. A considerable risk when trying to solve usability problems is going down the wrong track trying to come up with solutions that don’t truly address the issues at hand. Don’t worry, it’s not too much, and at the end of this article, you’ll find a spreadsheet that automates the whole process. Task analysis is the process of learning about ordinary users by observing them in action to understand in detail how they perform their tasks and achieve their intended goals. E.g. Record the task the user was attempting to complete and the exact problem they encountered, and add specific categories and tags (for example, location tags such as check out or landing page, or experience-related ones such as broken element or hesitation) so you can later sort and filter. Step 2: Have your prototype or product ready to Test . Borrowing from this logic, we have the following steps: Calculate the effectiveness of each solution. I'm keen on using this for a research study of mine so am looking for a specific reference to use. Well, that probably deserves an entire blog post, but let’s try to scratch the surface. How can we apply visual tools like sticky notes to work with the approach shown in this article? The purpose is to test the concept and build upon the initial framework (or scrap the idea altogether). It’s important to understand the limitations of this approach. Try and broaden the insight if it isn’t exactly identical with another, but is still strongly related. I took 1 hour + to get this reading done and being a product manager many insights on how to prioritize and come up with solutions came to my mind, and the best of all, involving all stakeholders: from CEO's, directors to developers. It's best to do this digitally, with a tool like Excel or Airtable, as you want to be able to move the data around, apply tags, and sort it by category. By clicking Accept Cookies, you agree to our use of cookies and other tracking technologies in accordance with our. Sometimes the solution is quite obvious—like correcting the placement of a UI component. It's not enough to simply present the raw data to decision-makers and hope it inspires change. Now that you have a list of problems, rank them based on their impact, if solved. Our updated table would look like this: In the example above, we have the following scenario: That’s it for now. A usability test will tell you whether your target users can use your product. Loved the double diamond approach (that I've seen around into the Design Thinking methodology I guess) and I'm not holding my breath to test it soon! It explains how to analyze and apply the results and what to do when confronted with budgetary and time restrictions. Simply put, define how critical the task is for the business or user by setting a numeric value to it. For example, instead of just “Avoid using a hamburger menu,” it’s better to state a specific solution, such as “Use a horizontal navigation and vertical tree menu.”. Now it's time to make sure you plan how you will analyse your results, with our top tips for usability testing analysis: Think about your deliverables. Research analysis. Calculate the ROI of the solution. It’s a basic percentage calculation. Quantitative data metrics for user testing include: Qualitative data is just as, if not more, important than quantitative analysis because it helps to illustrate why certain problems are happening, and how they can be fixed. Usability testing is often done in the primary stages of development. Good solutions are versatile! Using personality assessment tests with your current employees and sharing the results with them can help team members better understand each other, which is a win-win for all involved. Laborious as it may be, proper analysis is a worthwhile step that can maximise the value of usability tests. Conducting a usability test is one thing, analyzing it is something completely different. Such anecdotes and insights will help you come up with solutions to increase usability. The goal of usability testing is to detect any usability problems by collecting qualitative and quantitative data to determine the satisfaction of representative users with the product. Which one is more feasible? If you want to test on potential users, go where your potential users are. Once you've evaluated the data and prioritized the most common issues, leverage those insights to encourage positive changes to your site's usability. Applying the ‘double-diamond’ approach (diverging/converging problems and solutions), we can mix various user research data and use the methods above in any other project. Using these methodologies brought up the following observations from teams who used it in various projects: Especially when dealing with bigger studies, the issue prioritization keeps the team focused on what really matters, saving time and resources by reducing unwanted cognitive challenges like information overload, analysis paralysis, and decision fatigue; The connected end-to-end workflow keeps solutions more aligned with usability test outputs (because issues and solutions are paired), reducing the risk of implementing less-than-optimal solutions; We can easily apply this method collaboratively (in part or as a whole) using online tools. Carlos is a user experience strategist, researcher and strategist who creates useful, simple and pleasurable digital products and services. The result is shown in the image below: Including the fact that we left one parameter out (task criticality), the downside here is that you have to rely on visual accuracy instead of calculations as in the spreadsheet. In agile teams, where this subject is treated very seriously, it’s common to use business value and complexity, which lets us calculate the return on investment (ROI). These can be caused by a number of different factors, including decision fatigue and many types of cognitive biases. exactly what I was working on and this should help us a lot in making our own template better. In other situations, you may need to make your case to higher-ups at your company—and when that happens, you’ll likely need to draft a report that explains the problems you discovered and your proposed solutions. How standard are the technologies involved? And it’s hard to put into words … That said, it’s unnecessarily complicated to learn how to analyze usability test results if you haven’t done any testing yet. An inconvenient fact: usability testing will always uncover far more problems than you and your team are able to fix. Typically, usability tests aren’t complete at the conclusion without a list of recommendations (generic suggestions) and solutions (specific instructions). Before you start analyzing the results, review your original goals for testing. Before you start analyzing the results, review your original goals for testing. It’s important to identify the data you want before you begin testing. Planning for various eventualities can be a roadmap for the conversation. However, it’s not a walk in the park. Based on your objectives, decide the format of your test result compiling. If it’s still not working for you, there is also a visual approach where you can use Post-its and whiteboards. Copyright © 2014 - 2020 Hotjar Ltd. All rights reserved. Begin noting down insights (one insight per post-it) Add a label to each post-it (using your abbreviated labels) Place it down on your desk and prepare for the next one. Study sessions are full of rich feedback from participants, and it can be overwhelming to know where to start. double diamond from The British Design Council, eCommerce UX – An Overview of Best Practices (with Infographic), The Importance of Human-Centered Design in Product Design, The Best UX Designer Portfolios – Inspiring Case Studies and Examples, Heuristic Principles for Mobile Interfaces, Anticipatory Design: How to Create Magical User Experiences, Information Architecture Principles for Mobile (with Infographic), Evolving UX – Experimental Product Design with a CXO, Coliving Trends for the Remote Work Lifestyle, What Not to Do – The Beauty of Bad Product Design (with Infographic), Landing Page Best Practices (with Infographic), The first experienced by the participant one (P1), The second by the other participants (P2 and P3), 5: (blocker) the issue prevents the user from accomplishing the task, 3: (major) it causes frustration and/or delay, 2: (minor) it has a minor effect on task performance, 1: (suggestion) it’s a suggestion from the participant. Which solution is better? At this stage, we also have a good perspective on the usability issue landscape—the big picture that helps the team frame the high level problem and optimize during the following steps. Reasons for user testing. In the free Guide to Usability Testing, we divide the tests into four categories based on Christian Rohrer’s fantastic article: Scripted — These tests analyze the user’s interaction with the product based on set instructions, targeting more specific goals and individual elements. If you are an agile or design thinking practitioner, you know what I mean. (tree testing, hallway usability tests, benchmark testing) In order to simplify this approach, we had to leave one parameter out. It can feel redundant to create a report after a well-executed usability test, especially if you’ve involved the rest of your team from the start. Usability tests help you: 1. One of the easiest ways to get started with usability testing is session recordings. They may be things like: logging in, searching for an item, or going through the payment process, etc. Carlos. How to analyse a usability test. For example, you may find that several users had issues with entering their payment details on the checkout page. Adapting the model above to usability testing of the result is a four-step process: Data collection; Issue prioritization; Solution generation; Solution prioritization Maze also allows you to test your information architecture with blocks such as Card Sorts and Tree Tests. Regards. Which means that we do not have the final product to test. In addition, there is often a disconnect between research findings and actually implemented solutions. Identifying your main areas of interest will help you stay focused on the most relevant feedback. Identifying your main areas of interest will help you stay focused on the most relevant feedback. Your imagination is the limit! That's a very good question. Remind yourself of the problem areas of your website, or pain points, that you wanted to evaluate. On question : What do you think about having an approach that is less "issue" oriented (only negative feedbacks) ? Remind yourself of the problem areas of your website, or pain points, that you wanted to evaluate. Did you develop the tables used in steps 1-4 yourself or was this taken from the Lewis and Sauro reference? Based on the original design, you should deter… The difference is that it only serves to register the information, there is no calculation between them and the issues. The breathtaking How To Analyze Usability Test Results | Toptal Pertaining To Usability Test Report Template picture below, is section of Usability Test Report Template document which is listed within Report Template, usability test report format, usability test report template and posted at August 27, 2019. You're right, this method is suitable for collaboration while keeping solutions tied to problems. Similarly to issue prioritization, we need to prioritize solutions according to some parameters. Spot on! Read on for our best practices. The more severe the issue, the more effective its solution. The issues (i1 to i3) with their severities (4.95, 6.7 and 10.05), An indicator of 1 every time a solution matches (addresses) an issue, The effectiveness of each solution (4.95, 4.95 and 16.75), The complexity of each solution (1, 3 and 5) estimated by the team, The ROI of each solution (4.95, 1.65, 3.35). However, when determining a goal for task completion rate, context matters. Use a framework to document data. In an attempt to discover usability problems, UX researchers and designers often have to cope with a deluge of incomplete, inaccurate, and confusing data. The values may come from a simple linear sequence (e.g., 1, 2, 3, 4, etc.) Here’s a way to stay sane during the process. Prototype development is the first part of the process, regardless of the field that you specialize in. It can be a website that has limited functionality, a demo app or an interactive wireframe. If you want to follow this methodology, here’s a template (Google Sheet): https://goo.gl/RR4hEd. Divided into four distinct phases –discover, define, develop, and deliver– the double diamond is a simple visual map of the design process. However, usability reports are a strong method for communicating your results to your team and your wider organization in a clear, professional way, in order to build support for your research efforts. To summarize the steps: we started by collecting data, then we prioritized issues according to specific parameters. After the recommended changes have been decided on and implemented, continue to test their effectiveness, either through another round of usability testing, or using A/B testing. Keep a running tally of each issue, and how common it was. Meaning an "observation" approach (Positive+negative feedbacks). The risk is that there may be a disconnect between issues found and solutions identified. In order to master the obstacles mentioned above, we need efficient ways to handle our testing data while making sure we choose the most effective solutions for the issues found. Any site can benefit from some form of website analysis if the results are then used to improve it—for example, by reducing page size to increase overall speed or optimizing a landing page with lots of traffic for more conversions. Maze Discovery allows you to run research surveys, and collect user feedback early in the design process. This will allow the process to speed up and be efficient. Create the issue matrix by placing the sticky notes in the proper quadrant according to impact and frequency. The double diamond is exactly what we need to build a framework that will handle the usability issues and find ways to solve them. Awesome article! Use those areas of focus to create overarching categories of interest. Most likely, each category will correspond to one of the tasks that you asked users to complete during testing. May 10, 2017 - UX and usability testing analysis is a critical skill. Critical: impossible for users to complete tasks, Minor: annoying, but not going to drive users away. Tasks... Usability Evaluation (14) Reporting Usability Test Results. Thanks for the awesome feedback, Priscila! For instance, during the prioritization phase, the positive attitudes and behaviors of the users observed in testing are not included. In some cases, you may have the power to just make the changes yourself. In most cases, it’s sufficient to: A common approach for organizing usability issues, used by Lewis and Sauro in the book Quantifying the User Experience, is to plot the data as shown in the table below, with issues in the rows and participants in the last few columns. Watch the recordings, read the transcripts, and carefully go over your notes. Learn more in this blog. Once you begin reviewing the testing data, you will be presented with hundreds, or even thousands, of user insights. Without sacrificing the accuracy of the insight, you want to get through this step promptly. Typically, each usability problem has a grade of severity, influenced by some factors like: To prioritize, we need to follow these steps: Set the criticality score of each task performed in the test. Use Xtensio’s template to create a living document that you can collaborate with your team on to simplify the testing process. I did 2 things: a) went down to the gym and left a few flyers there (with the permission of the gym owner, of course) b) posted to a body-building Facebook group. Usability Testing. This is the cost-benefit relationship, calculated by dividing the effectiveness of the solution by its complexity. Thanks, Carlos! Consider how global the problem is throughout the site, and how severe it is; acknowledge the implications of specific problems when extended sitewide (e.g., if one page is full of typos, you should probably get the rest of the site proofread as well). Thanks ! Let’s go back to our spreadsheet, which now looks like this: According to this example, we should prioritize the development of the solutions in the following order (from the higher to lower ROI): solution 1, then solution 3 and 2. The first is a critical issue that should be corrected immediately, while the second is a minor issue that can be put on the back burner for some time in the future. Adjust the complexity of each solution by moving it on the horizontal axis (the more complex, the further to the right). How do you go about collecting data and analyze results? This makes usability testing an iterative process where each new cycle improves and refines the UX. Typically, a usability test involves extensive preparation and analysis, and is regarded as one of the most valuable research techniques. Hi Carlos, Thanks for this article. a few years ago I did website usability analysis for a body-building e-commerce shop. Nearly everyone I know (including me—of course) loves working with sticky notes and whiteboards, not only because it’s usually faster and fun, but also because it facilitates collaboration. This is an amazing article. Our method for conducting UX research and usability testing is to gather qualitative data by observing actual people (who match your Personas) as they conduct critical tasks on your website. Repeat the steps above for the remaining issues (quadrants 2, 3 and 4, in this order). Find the issue frequency (%) of the issue by dividing the number of occurrences by total participants. This is the ideal book for anyone involved in usability or user-centered design—from students to seasoned professionals. Consider your next steps before you plan how to conduct and present your analysis. 5m read . For example, a user who could not find a support phone number to call and another who couldn’t find an email address should be grouped together, with the overall conclusion that contact details for the company were difficult to find. Subscription implies consent to our privacy policy. He is a focused problem-solver and reliable team player, bringing together strategic thinking, creativity, and user-centered philosophy in his daily work. Compare feedback and statistics on success rates to evaluate the changes and confirm that they fixed the problem. I miss the "CHARTS" tab in the spreadsheet. Here’s a way to stay sane during the process. In this case, task criticality. Issues the user encountered while performing tasks, Comments (both positive and negative) they made, Bad example: the user clicked on the wrong link, Good example: the user clicked on the link for Discounts Codes instead of the one for Payment Info, Quantitative analysis will give you statistics that can be used to identify the presence and severity of issues. Afterwards, we generated solution ideas for those issues and, finally, prioritized them. Collecting, sorting, and understanding data gathered during user research and usability testing is becoming an increasingly common task among UX practitioners—in fact, it’s becoming a critical UX skill. Thanks for the feedback :-) Before you begin to analyze your results, think back to your objectives. Qualitative analysis will give you an insight into why the issues exist, and how to fix them. It helps identify problems people have with a specific UI, and reveals difficult-to-complete tasks and confusing language. So, with your test results in hand, how do you effectively create a usability report? Pro tip: make sure your statements are concise and exactly describe the issue. You are creating a list of problems with the website. Three usability issues experienced by three participants (p1, p2 and p3); The task ‘create a post’ appearing twice and assigned a, Each issue was assigned a value given its. Sort the data in your spreadsheet so that issues involving the same tasks are grouped together. The method above involves some (basic) calculations repeated many times, so it’s best to use a spreadsheet. How do you go about collecting data and analyze results? A good report should: Visit our page on reporting templates for more guidance on how to structure your findings. Note: We will need to use some basic math. In this chapter, we explain how to extract the data from your results, analyze it, and turn it into an action plan for improving your site. There are a number of points your plan should include, from the scope of the test to the number of testers, and we outline them all right now. One suggestion is to log this kind of data separately, and to use it along the way to complement and balance the findings as needed. Create the solution matrix by organizing the sticky notes according to each solution’s effectiveness and complexity: Brainstorm solutions for each issue, starting with the issues in quadrant 1 of the issue matrix (the ones with higher severity). The situation becomes trickier for those issues with non-obvious or many possible solutions. Each matrix is divided in four quadrants, indicating prioritization. When we understand how co-workers and managers prefer to communicate, the workplace becomes a more productive, comfortable environment. Aug 8, 2019 - UX and usability testing analysis is a critical skill. If you previously created user personas or testing groups, record that here as well. Taking any action to resolve the problems that have been uncovered is difficult if you’re not sure how to interpret or prioritise the results of your usability tests. It’s downloadable, and you can freely customize it to your needs. This method can measure improvement and determine the success of the results. If possible, please share yours thoughts after applying it :-). If they all encountered the same problem, then conclude that there is an issue that needs to be resolved. Finally, apart from usability testing, this approach can also be extended to other UX research techniques. Usability Testing also known as User Experience(UX) Testing, is a testing method for measuring how easy and user-friendly a software application is. We’re going to use the same divergent-convergent approach used to tackle the data collection and issue prioritization steps in the previous phase. Hi Fabien, AWESOME! When analyzing the data you’ve collected, read through the notes carefully looking for patterns and be sure to add a description of each of the problems. Once you begin reviewing the testing data, you will be presented with hundreds, or even thousands, of user insights. How to analyse usability testing results. The more severe the issue addressed, the better the solution. or something more elaborate like the Fibonacci sequence (1, 2, 3, 5, 8, etc. This article shows how a framework based on the double design diamond can alleviate both problems without further complication. How to analyze your data Mark additional issues that the solution may address—in practice, a single good solution can address multiple issues. Tags: Analysis, Analyze Usability Data, Usability Study, Usability Testing, User Research 0 I have noticed that there is a wealth of literature online on how to conduct a Usability Study, but I have yet to find anything to demonstrate how to analyze the data collected from a study. I've been reading this for more than 1 hour, give it all the attention and I consumed it lot better than other articles, thanks for sharing. In order to check the user experience, you need to have a product that you’re going to test. Usability Testing Essentials presents a practical, step-by-step approach to learning the entire process of planning and conducting a usability test.