Prioritizing & Reporting Usability Issues through Severity Ratings
Not all usability issues are the same. Rating usability issues by severity help prioritize and allocate the right resources to address the issues.
But…
What exactly is a Usability Issue?
A Usability Issue is anything that leads a user to an undesirable outcome. It might involve confusion around a particular term or a piece of content, navigation method, or users just not noticing something they should notice.
Examples:
- Behaviors that prevent task completion
- Behaviors that take someone “off course.”
- Actions that lead away from task success
- Expression of frustration by the participant, etc.
How does one identify a Usability Issue?
Observing what paticipants are doing, where they struggle, and how they succeed provides a great way to identify usability issues.
Enforce a think-aloud protocol in your study and observe for the following:
- Verbal expressions of confusion, frustration, dissatisfaction, pleasure, or surprise
- Verbal expressions of confidence or indecision about a particular action that may be right or wrong
- Participants not saying or doing something that they should have done or said
In addition to the above, if there is a possibility, observe their behavior, such as facial expressions and/or eye movements.
Why Severity Ratings?
Different types of usability issues have a different impact on the user experience, and Severity Ratings help direct our attention to the usability issues that matter.
How does one use a severity rating system?
Be consistent
Decide on a rating system and use it for all your studies. You will be able to make meaningful comparisons across studies if you use the same rating system.
Promote the system
Communicate clearly what each level means. The more the “non-usability” audience understands each level, the easier it will influence the design solutions for the highest priority issues.
Discuss and rate the issues
Make sure you have at least two observers taking notes during your usability studies. Invite them to a debriefing session, discuss the identified issues and rate them on an appropriate level.
Track the usability issues
Tracking the issues them more visible and makes it more likely that the issues will be remedied.
How many rating levels should one use?
A three-level rating system is generally sufficient. I prefer critical, major, and minor levels for both in-person study and heuristic evaluation.
How do you determine if a usability issue is…
Critical
Any issue that leads users to take the wrong decision and results in direct task failure. Critical issues impact effectiveness, efficiency, and satisfaction.
Major
Any issue that contributes to significant task difficulty but does not cause task failure. Major issues mostly impact the effectiveness and may impact efficiency and satisfaction.
Minor
Any issue that annoys or frustrates users but does not play a role in task failure. Minor issues impact efficiency and satisfaction.
Effectiveness, efficiency, and satisfaction are the measures of usability.
Who assigns the ratings for the usability issues?
If it is an in-person study, it is recommended that you work in collaboration with your product team, brainstorm on issues, assign a severity rating, and prioritize them. I generally do this exercise as a part of debriefing sessions post usability studies.
If it is a heuristic evaluation, allow evaluators to assign the severity ratings independently and then work together to discuss, agree upon, and finalize the ratings.
How does one report usability issues?
You can use any tool of your choice to document, rate, and report the issues. I prefer using an excel sheet with the following columns.
- Usability Issue
- Severity Rating (Critical, Major, Minor, Enhancement)
- Reasoning for Severity (Not applicable for enhancement)
- Design Recommendation
When reporting to the design management, I send them the summary of issues and a link to my excel document.
