Swapping survey stories

(Reading time: 1m, 44s)

Surveys are hard. There are so many ways to them screw up.

With new submissions to my survey of decision-makers at cultural organizations dripping in each day, I keep thinking of how I might improve it.

Developing feedback loops for clients is something I work on regularly. It’s curious that I’m so preoccupied with this one.

What questions am I not asking that I should be? Would it be better to drop a question or two to increase the completion rate? (As of today, it’s at 67%.) How might I improve the message to get more people to click through to the survey? On and on.

I remind myself that the purpose of this initial survey is just to question my assumptions about different kinds of museum leaders’ beliefs and what sort of research is being pursued at different kinds of cultural organizations. This survey is not meant to result in some definitive statement on what motivates audiences of cultural organizations to visit, join, or donate. We’ll need to go to the source — visitors, members, and donors — to understand that.

I am primarily using LinkedIn to recruit participants, so this is a convenience sample. That raises the question of whether the sample will be representative of the population.

But trying to reach every single decision-maker at a cultural organization in the United States is impractical and unnecessary. (I’m restricting the survey to the US, in part because I don’t want to deal with GDPR.)

Do you think the people I can reach on LinkedIn will have a different perspective on the questions I’m asking in this survey than people who are not on LinkedIn?

Using a survey panel may seem like an obvious option, but this is a fairly niche audience I’m trying to reach. So far I have just 30 people who have taken the survey, but look at who’s taking it:

Survey participants: Mostly executive directors

How many executive directors are participating in survey panels? I’m guessing not a whole lot, but there is no shortage of executive directors on LinkedIn.

And check out this thread by Alice Salisbury on the problems with survey panels:

Twitter thread on survey panels

If that doesn’t make you think twice about ever recruiting through SurveyMonkey again, I don't know what will.

All in all, given the purpose of the survey, I think we’re on the right track. But I always want to be open to the fact that there’s something I’m missing — We all have blind spots.

What do you see?

And what’s been your experience in creating surveys?

Have any surveys you’ve distributed led to insights that have changed the way you serve your constituents?

What’s the most surprising thing you’ve learned from a survey?

Let me know in a reply to this email. I’d love to hear from you.

Thanks for reading,

Kyle