Too Big to Tweet

Creating multiple concepts for design solutions is nothing unique. There’s often, if not always, multiple ways to solve a given problem. And at times, while teams may go to great lengths to internally evaluate, critique and eliminate concepts, they can end up in situations where they can’t clearly discern which solution is “best”.

When this happens, and no one on the project team either wants to make a recommendation, or feels empowered to, the tendency is to “let the user decide” by performing some sort of A/B split comparison study of the concepts.

On a project last year we were faced with just such a situation. It was compounded by the fact that the concepts were created by two different groups of designers, neither of which was willing to let their concept be eliminated by the team.

The project itself centered on individuals withdrawing money from a retirement account. These transactions are’t exactly straight forward. You don’t withdraw $1000 from the account and get $1000. There are costs and market factors that come into play. Because of this the design challenge centered on showing all of these costs and market factors to the user in a clear way that allowed them to configure their withdrawal to get them money they need and understand the impact on their account.

One concept was a very simple, wizard like set of static screens that displayed the inputs and calculations and allowed the user to move forward and backward within them until content.

The second was a more interactive representation of the calculations that allowed the user to click on various values, modify them, and see the impact on the other values immediately.

Enter the user-decided design concept thuderdome. Two concepts enter one concept leaves.

Due to time constraints the designers were required to very quickly turn around a representation of their concept based on a provided scenario. Both sets of designers created a set of wireframes with simple hot-spot based linking between them.

The user study was executed, and the wizard-esque solution came out on top. But the more I look back at the effort, the more I’m convinced that the study was flawed.

The argument was made that in order to accurately compare the concepts, they should be created and illustrated in the same fashion. In design we talk a lot about “fidelity” and we commonly portray design concepts at various levels that include: sketches, wireframes visualy designed comps, visually designed prototypes and working systems.

But it’s clear that the choice to portray the concepts as wireframes meant that the one concept, the simple wizard, was reflected in a way that more closely resembled the finished product than the more interactive concept. Wireframes just could not display the animations, realtime calculations and effects that the designers had envisioned.

Thinking about how we compare concepts, I feel that it’s more important that we look at fidelity on two scales: visual and interactive/behavior.

In comparisons on the visual scale, it is important to render the concepts in the same way. Sketches compared to sketches, visually designed screens to visually designed screens.

On the interactive/behavior scale however, it’s more important that the concepts reflect the intended richness of the interactions equally with regard to the finished product. 

The problem with the project above was that, for one concept, wireframes reflected the behavior of the system for the given interactions in a way almost identical to what was intended for the final product. But the limitations of wireframes meant that for the second concept, the behaviors were only about 15-20% of what was intended.

The study relied on users making accommodations for the fact that what they were evaluating was not the finished product, but in one case what they were looking at was far closer to the finished product than the other. The comparison was not a fair one.

As we go forward, it’s valuable for us to start letting go of the hard-line distinctions between mediums like sketching, wireframes and prototyping. After all, I know plenty of designers who are faster and more comfortable at “sketching” in drag and drop wireframing tools. And there are more and more tools that allow designers to turn wireframes into interactive prototypes.

What’s more important than medium here is message. What aspects of the solution do you need to communicate, and what’s the best way to do it?