As I mentioned in the previous post, the client was planning on engaging a top-tier firm to perform the heuristic evaluation. The quote they got gave them a bit of sticker-shock. Even to me (a salty vet), 30K seems steep for a heuristic review – especially when you consider they weren’t providing any design on the tail end of the review.
My take on this was that heuristic reviews need to be taken with a pretty sizable grain of salt. Studies have shown that the reviews have ~50% “hit” rate on issues they identify. This means that about half of the issues identified by experts looking at a system are actually identified as issues by user testing. In addition, “experts” bring with them personal biases and a pretty deep familiarity with technology which is hard to forget, making them a fairly non-representative user in most cases.
The backbone of most heuristic reviews is still the standard set of “heuristics” or dimensions of analysis identified by Nielsen and Molich back in the early 90s. Since the field of usability has matured, almost anyone in the field is familiar with them and how to employ them in a review. Given the widespread knowledge of the principles and methods coupled with the relative inaccuracy of expert reviews, I’m forced to question how much value is derived from engaging a top-tier firm.
Certainly, there’s a bit of “nobody ever got fired for buying IBM” in the equation… However, a smaller firm would hit on the bulk of the same issues that the top-tier firm would. Even if the extra level of expertise guarantees the top-tier firm will identify 10% more issues, that still only means a 5% increase in “hits.” And, since the process has no design interpretation built into it, I fear the extra expertise (and then some) will be lost in the translation.
My recommendation to them was to engage a smaller firm to conduct the analysis and use the savings to have the same firm provide design services and work with the developers to ensure the system gets built as designed. That way, at least the design gap is closed and they know they will realize as much value as possible from the heuristic review.