Focus Groups vs User Interviews: Which Is Better for Uncovering Audience Insights?
I’ve been doing some research on brick-and-mortar membership organizations lately. I found something that may be helpful to you in a presentation describing research into membership development for The Art Institute of Chicago.
For today, let’s focus on a single slide:
Critiquing research from a presentation that you didn’t attend isn’t all that fair. Bullet points like these in a presentation are usually just mnemonics for the presenter. There's usually more data that we can't access because we weren't there.
But the main take away here is that focus groups provide little value for audience research — and I believe that's true in almost every case, no matter what the supporting evidence may be.
So before we return to the content of this slide, let me climb up on a soapbox the size of a mobile home and explain why focus groups are a bad idea:
The group is focused on finding a leader
Social dynamics influence what people will (and won’t) tell you. Shocking, I know.
Just because someone is persuasive or assertive in a group doesn’t mean they are knowledgeable or have any greater insight. But we are biased to believe that people who are assertive and confident must have something valuable to say, so people in a focus group will defer to those who dominate the conversation. Introverts are drowned out.
It's also easy for us to believe that the people gathered in a focus group are all equally invested in the exercise — but they’re not. They’re just not as committed as you would like them to be, so if one or two people in the group are vocal and seem to be moving the discussion toward some conclusion — any conclusion — participants will mentally shrug and go along. It's easy to mistake that for consensus.
You just bought tickets to Research Theater
Focus groups look and sound very official. Yes, you can interview people individually and come up with useless or misleading data, but it’s easier for worthless data to masquerade as valuable research when it's called a "focus group" and brings to mind two-way mirrors and people in lab coats.
You can wind up in a situation where someone has dumped a bunch of college students into a room to talk about stuff they like. These are people who are unlikely to buy your expensive product or become a member of your organization, but, so long as it’s called a “focus group”, the CEO will knit his eyebrows and nod approvingly.
He may as well just drive them to the gas station and give them some beer money.
MOAR GROUPS MOAR FOCUS
No, more people does not lead to more insights. Qualitative research isn’t like analytics. You don’t need vast numbers of people to get meaningful results. Just as with user testing, five individual interviews can produce all the results you need. Beyond that, you get diminishing returns.
And focus groups don't save you money. In fact, they can be more expensive than individual interviews. If one or two individual interviews go bust, you can throw them out and replace them. If one focus group goes bad, you have to replace the whole group. This creates more subtle pressure on researchers to gloss over worthless results from a focus group.
Did I mention social dynamics?
People who haven’t read a book since 1982 will tell you they love to read — and they actually believe what they’re saying is true — because we all want to look smart in front of other people. Yes, people can mislead you in a one-on-one interview, but focus groups magnify the urge to present oneself to the group in a flattering light. A group of strangers will compete with each other to gain approval.
You may as well cover the conference room floor with sand and pass out swords.
"What would you do if you had a billion dollars and could freeze time?"
I suspect that if a researcher believes that focus groups are a good idea, they're likely asking the wrong questions as well.
And this is where we return to the slide, which says:
Members participating in this research did not explicitly state a desire for AIC membership to change significantly.
That’s a flag!
If members aren’t stating a desire for change, that suggests someone is asking them what changes they would like to see.
They don’t know what changes they want to see.
Valuable interview questions are tied to facts — not speculation. It’s not realistic to expect people to dream up new programming or product ideas on the spot. They’ll unconsciously monitor you or the people around them for clues and come up with whatever sounds smart or seems like the right answer. (If you're interested in reading more about this, read The Mom Test by Rob Fitzpatrick.)
So, it’s no surprise that focus group participants struggled to come up with changes or new benefits for the museum’s membership.
What did they come up with?
The group concluded that (emphasis added):
Membership should remove the hassles of visiting while making me feel good about what I’m supporting.
Now, I wasn’t present for this presentation, so I’d like to give the researchers the benefit of the doubt and assume that there’s a lot we’re not seeing just by looking at the slides.
That said, purely for the sake of argument, let’s pretend that this was presented as an insight that was valuable in and of itself.
That would be another penalty.
Because you have to wonder:
Who doesn’t want to have the hassles of visiting your place of business removed?
Does anyone want to feel bad about contributing to your organization?
In other words, did the focus group really draw out any insights into members’ needs?
Presumably many museum visitors, not just members, would like to have the hassles of visiting removed — whether that be long lines or a poor experience purchasing tickets online — and many people would like to feel good about supporting the museum by visiting.
If focus groups are Satan’s spawn, what should you do instead?
You've probably guessed by now that I'm going to suggest interviewing members or customers one-on-one.
Interviewing can be challenging, but if you can stay focused on the person’s past experience and uncover their habits, you can start to identify potential opportunities for improvement.
Ridiculously specific questions are a way to get accurate information from people:
Where were you when you became a member? Who were you with?
Tell me about the last time you used the member's lounge.
Think back to your last visit — did you drive to the museum or take the train or a cab?
What did you see while you were there?
What did you do after you left?
And always ask why.
Questions like these don’t suggest to the interviewee that there are any right answers. You don't seem to be leading them anywhere in particular.
But they can help you find something unexpected about the experience of becoming a member through your website or at your location. You may learn that free parking could be a valuable perk to members if they’re having to pay for a cab each time they visit — or you might look into a partnership with Lyft. Maybe a members’ lounge or improvements to the cafeteria are in order if people are leaving to find food …
Interviewing people is a topic for another day. The point is that by interviewing people individually about their actual experiences, the hypotheses you put together will be supported by facts.
The killer isn’t going to do your job for you, detective.
There's no shortcut where the interviewee comes up with a great answer that no one has to translate or test. A customer interview gives you clues that you can use to build a more complete picture of the customer experience, which can then lead to ideas to improve your business. Going into a focus group and expecting people to come up with ideas to improve your business is like a detective walking into an interrogation room expecting a confession. It can happen, but it’s more likely he’ll only find clues to help his case — not solve it that day.
Bottom line: The next time you discover a focus group underway at your organization, just pull the fire alarm because the only meaningful data you’re going to collect will be whether people can safely find the exits.
Thanks for reading,
This post was originally sent to the SuperHelpful mailing list. Sign up to receive future letters.