A reader asked on Twitter:
Any recommendations on how to combine Shape Up and usability testing? When is the good timing for it? Should it be included in the cycle? Who should take responsibility for it?
To answer this, I first divide usability problems into two kinds:
- Perceptual: "They couldn't figure out what to do next", "they couldn't find the feature", "they didn't know they could click that button..." etc.
- Domain-specific: "We need a way to jump back here because in their workflow this happens..."
In general, usability testing only catches type 1 perceptual problems. Because in those tests you take people out of the real world and assign them tasks. Usability testing doesn't catch domain-specific problems because they only come up in real life use.
I consider type 1 perceptual usability to belong within a designer's education and training early in their career, not as part of production projects. Once you know the basics, you can apply them again and again to any project. For that reason I've never done that kind of usability testing in projects.
The type 2 domain-specific problems are very different. Those about understanding what matters to make the design work so that it does the job for the specific purpose in the specific situation. To do that, you have to understand the world around the design and the progress people are trying to make when they use it.
If you try to do that with testing, it's already too late. You need to get an understanding of the problem you're solving at the earliest stages, before you've built something to test. It's about forming a better hypothesis.
In terms of process, I put type 2 domain-specific usability in the shaping phase. Getting it right first means doing the research and/or living the problem so you understand what to do. Testing at this stage means plugging early, cheap prototypes into real life to see what happens.
For example, the first version of the Hill Chart in Basecamp was a Numbers spreadsheet I built that generated the graphs. We used the spreadsheet in real projects to become confident that it was actually useful before building it with software.
To catch and solve domain-specific usability problems, the most important thing is understanding the problem and the desired outcome first, independent of the particular design solution. If you deeply understand the space the solution needs to fit into, you can iterate without third-party testers.
This approach is the most cost-effective way to catch the majority of usability problems. The designer should internalize perceptual usability as part of their skill set — they shouldn't need to learn the same lessons over and over. The big domain-specific issues should be caught and prevented early with research and by testing cheap prototypes in the shaping and pre-shaping phases, before placing a bet and building the whole thing.
Catching the last 5-10% of problems before release is somewhere between prohibitively expensive and impossible. It's much better to ship, get the product into real life, then prioritize what to do in future projects based on feedback. A regular cadence (eg. six-week cycles) and clear path from problem to shaped improvement to execution enables this.