Blaming the hammer
[Reading time: 3 minutes, 37 seconds]
Last week, I spoke with Dr. Ari Zelmanow by phone about qualitative research for museums.
Ari is a former police detective who earned his doctorate studying human learning and decision-making. Today, he helps businesses better understand their customers to improve their products and services.
I contacted Ari because I wanted to get a fresh perspective on what I’ve learned so far in my research into the museum world. I didn’t record the call, but I did take notes, and Ari was kind enough to let me summarize his comments here.
First, I explained the patterns I had heard in conversations with decision makers at cultural organizations — in particular, museum’s heavy reliance on surveys and focus groups for audience research along with a concern among some leaders about the difficulty of figuring out what their museums should look like in five years.
(Quick aside: Ari and I wound up talking a lot about the limitations of surveys and why interviews are so valuable. You can read about why focus groups are so often inferior to one-to-one interviews elsewhere on the SuperHelpful blog.)
I tried to not lead Ari toward my own conclusions. I just shared how museums are handling research today and the decision makers’ concerns as they framed them. Then, I asked him how he would help an organization in that situation.
Ari pointed out that the problem with surveys is that you won’t know if you’re asking the wrong questions. You don’t know what you don’t know, and surveys won’t help uncover that. Surveys are often an expression of your assumptions about your audience. Moreover, they tend to be self-selecting. Only certain kinds of people will take the survey; You’re unlikely to learn from new audiences — people who aren’t moved by the museum’s offerings as they stand today.
(During the call, I tried to keep my mouth shut and was glad Ari couldn’t see me nodding so hard I nearly injured myself. I kept thinking of museum folks who told me things like, “We keep surveying people — the surveys never turn up anything new, but we keep doing it anyway, just in case.” )
He said he would tell museum leaders to segment their audience and interview people from those different segments regularly. (A few that spring to mind: Members, lapsed members, parents of children who have aged out, grandparents, tourists, people who would never set foot in a museum — there are lots of ways to segment an audience.)
Then, he suggested that those interviews will lead to new insights and hypotheses that can be tested at scale by properly segmented surveys.
Of course there’s a place for surveys — I run them continuously for clients today! Using them to support interviews makes perfect sense.
I’m always talking about triangulating and never relying on a single source of data, but I think over the course of talking with so many organizations that rely so heavily on surveys, I started to slowly devalue them.
I was becoming biased against surveys in considering all the ways I might help museums, rather than reminding myself that they’re just another tool in the toolbox.
If someone uses a hammer for everything from nailing in screws to chopping wood, you can’t blame the hammer when everything winds up bent and pockmarked.
Another thing that struck me in our chat was Ari’s sense of urgency. When we talked about leaders’ concerns regarding longer term planning, he said museums should start interviewing regularly NOW — as in, yesterday:
“In 5 years, it’s too late.”
It’s true — you don’t get the time back.
It’s the same thing with analytics data. You can clean up all your processes and data sources and start documenting user behavior today, and soon you’ll wish you had started long ago because you can see the value in longterm trends analysis.
Data is like shell cordovan — it’s nice when it’s new, but it becomes beautiful with age:
The shoes on the right looked like those on the left when they were new.
Fresh data is like a new pair of shoes — the potential is there, but it takes time to really build the relationship.
(So maybe I have a weird thing for leather shoes. So what?)
The point is, once you have your data in order, some patterns can’t only be uncovered over time, and the waiting can be painful.
Finally, I told Ari that many museum folks worry that they can’t afford this kind of research.
He asked why they don’t use the budget they have in a different way. Qualitative research — interviews and contextual inquiries — would reduce the amount of variance in the organization’s surveys. They could save money on the quantitative research they’re familiar with and get more valuable data from qualitative studies.
What do you think of that?
I’d be moved by the thought that I could stop running the same surveys over and over, getting the same results, and instead begin to find ways to reach new audiences — and perhaps understand which audiences might truly be a dead end and know where to focus my efforts in the future.
What about you? Hit reply and let me know.
Thanks for reading,