Looking for long-term impact from a short-term engagement

(Reading time: 2m, 21s)

Decision makers at cultural organizations often talk about how they wear many hats. Every day is a balancing and juggling act. It’s not that every person I speak to feels overwhelmed, but what often gets prioritized are the needs of the moment, which doesn't leave a lot of room for new thinking.

One of the ways I try to help is by shaking up that pattern.

For example, one of the services I’m working on is a Live Website Evaluation. A live eval has two main parts: A series of user tests on the client's website followed by a half hour video call with me, during which we discuss things they do to improve the site to improve visitors’ experience and facilitate more valuable interactions. I try to make recommendations that are realistic. (This often means I have to bite my tongue when it comes to significant problems stemming from the use of third-party software. Hello, Blackbaud.)

Before the eval, clients fill out a short questionnaire, which I use to inform user testing and my own review of the site. I ask things like:

  • Which of the following website transactions generate the most revenue for {org}?
  • On average, how many transactions does your website currently generate each month?
  • What is the number one thing you believe first-time visitors to your website may struggle with?

Questions like these touch on units of measure that many museum folks don’t often prioritize and surface hypotheses (or assumptions) I can test for them. As in, getting latent ideas out of their heads and actually testing them with real people. It’s often the first time that’s happened.

The questionnaire leaves no room for their objections, which might go something like, “Things like getting visitors to donate or become a member are important, but so are visits to our blog and people getting directions to visit our museum.”

Fair enough. It’s not that those things don’t have value. It’s just that the purpose of the evaluation is to shake up what these decision makers usually prioritize and measure. They’re comfortable measuring website visits and pageviews. They’re less familiar with measuring the impact of usability and content on revenue.

The service promises to uncover practical ways museums can improve their websites, but what interests me most is a chance to expose decision makers to a different — perhaps more hard-nosed — way of prioritizing design decisions for a short time.

I suppose the question is: To what degree can a service like this influence or change a person’s thinking?

An evaluation takes half a day of work on my part. For the client, it’s just half an hour, plus the time to fill out the questionnaire and review the testing summary.

So far, I’ve tested the service with a handful of museums. (It’s changed a bit over time, so it depends on how you count.) Two have implemented some changes based on the recommendations. One has made major changes based on the feedback. The results are less clear with two others — too early to say.

If the goal is to do more than help museums make better design decisions, this service probably isn’t enough. If the goal is to expose museums to new ways of prioritizing design decisions, I'm probably way too subtle.

What do you think? Smack the reply button and let me know.

Thanks for reading,


Kyle Bowen