Stop chasing opinions

What do you think when you see this slide:

Whose opinions are missing from your planning and decsion-making processes?

That’s a slide from a presentation by a survey company. I watched the presentation the other day and had to go outside for some fresh air.

I’m finding that visitation-based membership organizations rely heavily on surveys for audience research.

Surveys are compelling for these organizations because they’re easy to produce and they feel true. They spit out these numbers and statistics that just feel so — science-y. And science is true, right?

It’s understandable. Surveys are affordable and nothing feels better than going to your boss or your board with a tidy report full of charts.

So organizations come to rely on surveys.

Fo many visitation-based membership organizations, surveys have another appealing quality — They feel more inclusive and seem to account for more perspectives.

Museums value inclusivity. Inclusion and educational impact are ideals they strive for. They want to reach more people from diverse backgrounds — ideally, the people visiting their museum would be a perfect reflection of the demographics of their city.

Combine these two things — the ease and truthiness of surveys along with a penchant for inclusion — and you get this:

Opinions again

You get people chasing opinions. And not just anyone’s opinion — everyone’s opinions.

At one point, the presenter in this webinar explained how wonderful it is that they use a 10-point scale for many of their survey questions:

“Everyone can find a number that works for them.”

In the world of user research, Likert scale survey questions — “rate x on a scale of 1 to 10” — are largely viewed as an unreliable data source. Just ask Eleanor Roosevelt.

I’m not going to rehash all the problems with these kinds of survey questions.

What grabs my attention is that surveys may be a real weak point for cultural institutions. Their desire to be inclusive is picked up on by research and marketing companies who shape their products to match that desire. They create survey tools and focus group services that promise to get everyone’s opinions.

There are two problems with getting everyone’s opinions:

  1. Some people matter more than others: The opinions of “interested nonmembers” or (worse) city “residents” — people who have no relationship to the institution — shouldn’t have the same value as members, lapsed members, or at least people who have visited your organization. Many of these surveys deliver little to no segmentation, which makes for beautiful charts that can be quite misleading.

  2. Opinions are like armpits. Everyone has them and everyone thinks everyone else’s stink: This is user testing 101. Chasing what people like is waste of time. It’s far more valuable to uncover what different kinds of audiences need to achieve goals that benefit both the user and the organization.

I’m not saying surveys don’t have a place. But they’re often incomplete with supporting evidence like interviews and analytics assessment.

Do you run surveys to gather audience data? Have you tried other ways of studying your audience? If not — why? What’s prevented you from doing so?

Hit reply and tell me what you’ve found.

Thanks for reading,

Kyle

Kyle Bowen