Testing the limits of remote user testing

(Reading time: 4m, 44s)

A few weeks ago, I compared ways in which the Long Island Children’s Museum (LICM) and the Please Touch Museum (PTM) communicated value to prospective visitors through their websites. I especially admired how PTM’s value proposition — “We build brains” — spoke to the unique value of the museum, suggesting an audience-focused benefit that visitors might not find elsewhere. LICM's headline "Explore Play Create" certainly has its appeal, but those are activities that can be pursued in many different contexts — it doesn't communicate the museum's unique value quite as well.

A number of you all responded that that letter, and value propositions are a recurring theme here — So, I thought I’d try a short experiment to see whether those value props resonated with first-time visitors to those museums’ websites.

In other words, would casual visitors prefer one statement over the other — or even read them at all?

Boyfriend meme: museum value proposition edition

So, I ran eight remote user tests on the websites to see how first-time visitors assessed and interacted with the content.

Activities vs impact (a little more background)

Many (most?) cultural orgs present a menu of activities on their website's home page. There isn’t often much or any space devoted to communicating the why behind those activities.

Does it matter?

In the case of children’s museums, for example, are visitors persuaded by more high-minded or outcome-focused messages — or will a menu of today’s activities suffice?

To be sure, there are other ways to find out. It would be better to look at the question from several perspectives — studying behavior over time in relation to messaging through analytics, interviewing different audience segments, and user testing to ensure technical issues aren’t skewing behavioral data.

But if I want to compare these two websites today, remote user testing is the quick and dirty approach that will let me scratch at this question that’s been itching at me.

Acknowledging some limitations and flaws

I’m working with a pool of people who have volunteered to participate in user tests — some turned out to be, or have been, parents of small children, but not all of them. So, they aren’t all perfectly representative of the people who might be evaluating these websites normally.

That said, it usually does not matter a whole lot who completes a user test because you’re normally testing functionality, not value propositions. If a children’s museum’s online checkout is a pain in the butt, it doesn’t matter if the person testing has a child or not — a college student with no children is just as likely to uncover usability issues as an older adult with children.

In the tests I ran, I asked participants to imagine they were the parent of a small child and then answer questions like:

  • Find out more about what this organization can offer you and your child by exploring the rest of the site (beyond the home page) for a few minutes. How would you go about deciding whether to visit or become a member of the museum? Narrate your experience as you go.
  • What makes this organization different from others of its kind? (If you are unfamiliar with this kind of organization, describe what stands out to you as being potentially unique.)
  • Based on what you've seen and read today, what makes these two museums different?

(How artificial is that last question, asking people to compare two different children’s museums? How often are caregivers in the real world trying to decide between two different children’s museums? I bet it’s pretty rare — People are far more likely to be comparing a single children’s museum to the prospect of staying home and playing in the playroom. Still, I have to make the itch go away, so I ask.)

I’m not testing functionality or usability — I was curious to see how people would evaluate the organizations in terms of what they promised visitors. I think many visitors do ask themselves these questions when they visit organizations’ websites — they’re asking things like, “Is this going to be worth it?” and “What’s in it for me/us?” — but there is an admittedly artificial quality when an outsider poses those questions.

Behavior by the numbers

First, I wanted to see if people would even read the statements on the respective websites that most resemble enduring value propositions — for LICM, that was “Explore Play Create” and for PTM it was “We build brains.” Then, I wanted to see if people would cite those value props in some way when asked to identify what made the organization unique. If someone read the value prop, would it in some way color their view of the museum for better or for worse?

Seven of the eight participants actively read LICM’s value prop — that is, they actually stopped and read it out loud as they explored the site. Just one seemed to scroll past the statement without reading it.

Only three participants actively read PTM’s value proposition — the remaining five scrolled past it, but didn’t pause to read it aloud. I think this may have more to do with the placement of the PTM statement, which is further down the page. LICM’s headline is the first and perhaps most prominent thing visitors see.

(Both statements are super short and prominent — so, it’s possible that a participant could read them in a more passive way, without actually reading them out loud, and the statements could still color their assessment of the museum in some way.)

Results: The itch hasn’t gone away

PTM’s value prop and similar mission-oriented content clearly made an impression on two participants. They read the statement, remarked on it, identified it as something that made the museum unique. That’s two out of the three who actively read the value proposition — “We build brains.”

When asked what made the museums different from each other, half the participants identified LICM’s live theater offering as a differentiator. Three of those four people did not actively read PTM’s value proposition.

Of course, I want to conclude that, in the absence of an encounter with an enduring value proposition, visitors will latch onto different activities as a differentiator. And if they do encounter a mission-centric value proposition, they’ll latch on to that as a compelling differentiator.

But I can’t draw that conclusion from this little experiment because there’s all kinds of variables at play and the results just don’t bear that out.

I’ll need to do more and different kinds of research to answer the question.

Will you share this rash with me?

Is your organization’s home page more or less a menu of activities? Have you experimented with including more enduring value propositions? Have you tried to measure the impact of those changes on visitor behavior?

Let me know in a reply — I’d love to hear how it’s going.

Thanks for reading,

Kyle

How helpful was this letter? Share your feedback.

P.S. After sending today’s letter, my wife told me that the “distracted boyfriend” image I included could be seen as objectifying women. I had thought that presenting something as nerdy as museum value propositions in the context of a ridiculous meme would be funny.

So, in an effort to restore balance to the universe, I present the “Girlfriend Distracted by Museum Value Propositions” meme below.

girlfriend distracted by museum value propositions