Usability Severity Ratings: Calibrate Findings Before Prioritizing Fixes

A usability severity rating describes how much a problem disrupts a person's task. To make ratings useful, describe the same problem to every reviewer, agree what each level means, and resolve disagreements against the evidence. Keep the decision about when to fix it separate: an easy repair is not automatically a severe problem, and an expensive repair is not automatically a minor one.

Severity assessment follows observation. If you are still deciding how to collect that evidence, start with the usability testing process. This guide uses a fictional equipment-loan service, Alder, to show how a team can discuss one ambiguous confirmation without turning a rating into a vote about personal preferences.

Describe the problem before choosing a number

In Alder's worked example, a borrower selects a camera kit and saves a loan request. The interface says “Request saved.” The service has stored a draft, but the wording can suggest that the request was submitted. The consequence under review is a borrower leaving without sending the request, not merely disliking the word “saved.”

Write an issue record that another reviewer can understand without attending the original session. Include the person's goal, the starting state, the visible action and response, the misunderstanding or obstacle, and its effect on the task. Attach a permitted observation reference and the relevant screen version. Keep an observed action distinct from the team's explanation of why it happened.

A useful record might read: “After saving a camera-kit draft, the borrower treats ‘Request saved’ as completion and leaves. No loan request has been submitted.” That is a synthetic example of issue wording, not a quotation from a participant. “Confirmation is confusing” would be too vague to rate consistently.

Merge records only when they describe the same underlying problem. Several notes about this one confirmation are not necessarily several affected people. The research synthesis workflow explains how to preserve source references and contradictory evidence before turning observations into a finding.

Give each rating an observable anchor

Choose a convention that matches the consequences of the product's tasks. Nielsen Norman Group's severity-rating guidance considers frequency, impact and persistence. In other words, discuss how often the problem arises, how difficult its consequences are to overcome, and whether learning the interface makes it go away.

For Alder, the team uses the following working categories. These are example anchors for discussion, not universal release rules. They refer to the affected task, so a visually small issue can still be serious.

Keep “not a usability issue” and “insufficient evidence” outside those four levels. A broken prototype connection may need repair before the design can be evaluated. An uncertain report needs investigation. Neither should be quietly assigned Low just to empty a review queue.

If your organization already uses a numeric scale, retain its definitions and map these questions to it. Do not mix one team's “1 means urgent” with another team's “1 means cosmetic.” Display the meaning next to the label wherever ratings are discussed.

Rate independently, then examine the disagreement

Give reviewers the same issue record before showing anyone else's rating. NN/g recommends independent ratings when evaluating a shared problem list. A short written rationale makes the later discussion more useful than asking people to announce a score around the room.

Suppose Alder's first reviewer chooses Medium because a borrower can reopen the draft and submit it. The second chooses High because the borrower believes the task is finished and has no reason to reopen it. Their disagreement concerns the recovery path, not their tolerance for untidy copy.

Inspect the actual state after saving. Is “Draft” visible? Does the interface say that submission is still required? Can the borrower see the request in a draft list, and would they know to look there? These questions test the assumptions behind the two ratings.

The Alder review pairs the ambiguous “Request saved” state with a proposed “Draft · Not submitted” state, while keeping the issue rationale beside both screens.

If the available evidence supports a mistaken completion and no recognizable recovery, High fits this team's convention. If a clear next action remains visible and people recover independently, Medium may fit. Record the evidence that changed the judgment. Do not average the labels into a supposedly precise score while leaving the recovery question unanswered.

When the evidence is incomplete, preserve the disagreement and assign a specific follow-up. For example, inspect the draft-list entry and confirmation together, then observe whether the revised wording communicates the remaining action. A documented uncertainty is more useful than forced agreement.

Keep frequency and confidence visible

“Several people mentioned it” is not enough to establish how often the problem occurs. Count unique relevant task attempts, record who encountered the state, and separate repeated comments by one person from independent observations. A small qualitative round does not establish the population frequency of a defect.

A serious misunderstanding seen once still deserves attention. Conversely, a frequently noticed alignment defect may have little effect on task completion. Use frequency to explain exposure, not to erase the consequence. If only a certain role encounters the problem, preserve that context rather than averaging it away.

Keep evidence confidence in a separate field. “High severity, limited evidence” tells the team that the potential consequence matters and needs checking. “Low severity” would incorrectly suggest that the consequence itself is small. Also record whether the difficulty returns on each attempt or disappears after the person learns the interaction.

Decide priority without rewriting severity

Severity describes the problem for the affected user and task. Priority is the team's decision about action, considering severity alongside reach, dependencies, delivery constraints and other risks. Keep both in the issue record so a scheduling decision does not rewrite the finding.

Alder might fix its confirmation wording immediately while a larger request-status change needs coordination. That does not make the wording issue minor merely because it is quick to change. Equally, a Low visual issue may be fixed during an adjacent component update without being promoted to High to justify the work.

For a deferred issue, name an owner, explain the reason, and state what would reopen the decision. Avoid a generic “later” column that loses the user consequence. Where a workaround is accepted, describe who can use it and how they will discover it.

Use the rating to focus the next design check

In a Mockitt prototype of the revised request flow, prepare the saved-draft confirmation, the draft list, and the submitted-request state. Keep these outcomes visibly different. The prototype can represent the proposed journey; the research record and severity decision remain separate from the drawing.

Ask a reviewer to explain whether the camera-kit request has been sent and what remains to do. A later participant session should observe the same task consequence that motivated the change, rather than simply asking whether the new screen looks clearer. Use a testing plan with defined outcomes to keep assistance and completion rules consistent.

Retain the original issue, the agreed rationale, the design version and the follow-up evidence. Marking a ticket “fixed” records an implementation action; evidence that the revised path resolves the misunderstanding supports closing the usability issue. That distinction keeps the rating useful after the meeting ends.