Cinema Arts Centre User Testing Results

A study of the CAC website's user experience

In this analysis, you'll find our recommendations for addressing user experience issues primarily related to:

  • directly communicating CAC's unique value proposition, and
  • organizing information and designing content to improve visitors' ability to complete revenue-generating tasks through the CAC website.

Background

  • What is user testing?

    User testing asks people who are unfamiliar with an organization to interact with its website or other digital products. The users narrate their experience as they complete a set of tasks. These can be done in person or remotely.

    User testing doesn't seek to survey individuals about their opinions. It doesn't seek to know about a person's aesthetic preferences, though these are sometimes expressed. Instead, user testing examines the functionality of a design, not so much how it looks. It uncovers how a design system — a website, for example — matches up with users' expectations and mental models.

  • Why is user testing valuable?

    User testing forces designers and stakeholders to experience what it's like to interact with a company through its digital products for the first time.

    The people who work together to develop websites — designers, executives, project managers, and so forth — will naturally become so familiar with a product that it becomes all but impossible to see the work with fresh eyes. Moreover, people tend to become invested in a project, which leads to (mostly unconscious) biases.

How we tested

For CAC, we conducted remote user tests. Users were given a set of tasks to complete and asked to narrate their experience as they went. They use the same computers or mobile devices they use at home, and their screens and microphones are recorded throughout the process.

We conducted nine tests for CAC, resulting in 90 minutes worth of video. You'll see some of these clips later in this report.

user-testing-diminshing-returns-curve.gif

Diminishing returns after 5 user tests of the same task.

Source: NN/g

Nine tests may seem like a small sample, and if this were a survey that would be true.

But for user testing, five users is recommended and, in fact, more than 6 users testing the same set of tasks tends to result in diminishing returns. (We used nine tests because the service we use generated that number of tests.)

What we tested

Our goals were to:

  1. Get a sense of first-time users initial impressions of the site, as well as their understanding of CAC's purpose and unique value.
  2. Determine the usability of the website's navigation and the site's content design and organization.
  3. Test the how easy/difficult it is for people to sponsor CAC on behalf of their business.

Here are the testing tasks:

  • Take 10 or 20 seconds to look through this page and share your first impressions.

    First impressions are often based on visual design. A site's usability — i.e., people's ability to complete transactions — is critical, but visual design often tints people's perception of a site's usability. As you prioritize usability improvements, it's good to keep in mind that good visual design can be a valuable aid to perceived usability.

  • Based on what you see, describe this organization and what you can do here.

    This question contributes to our understanding of how effectively the site communicates CAC's identity and value to visitors.

  • Now, without clicking any links, review each of the main navigation links one by one. Briefly describe what you would expect to find if you were to click on them.

    Here I'm testing the site's taxonomy. Does the language used to organize content match up with new users' expectations? This is important because navigation is the website's road signs. Poor signage will lead users down the wrong paths, which can lead visitors to give up before completing important tasks, such as donating.

  • What strikes you about this website? Narrate your thoughts as you spend a few minutes reviewing the site.

    At this point, users will have formed ideas about CAC and the website. While user testing is often highly directed, in this case I wanted to see what questions users would develop independently of my direction, and how they would use the site to answer those questions.

  • Imagine the following scenario: You're a local business owner who would like to support this organization. What options are available to you? How would you go about supporting the organization on behalf of your business? Do that now and narrate your experience as you go.

    While I'm sure sponsorships from local businesses are not as valuable or common as other tasks on the website, such as membership renewal, I suspected based on my own experience that this process on the site could benefit from improvement.

  • What makes this organization different from others of its kind?

    By now, users have spent some time using the site. I wanted to see if CAC's purpose and value proposition was apparent.

  • What, if anything, would you change about this site?

    Questions like this can sometimes lead nowhere in testing. Some users will give you opinions based on their personal taste, which is not the purpose of testing. Sometimes, though, the question can lead to a summation of what users find most challenging about the experience of using a site.

Results & Recommendations

From the original 90 minutes of testing, we've compiled 26 minutes of highlights below.

We cut portions that were redundant, could be communicated more succinctly in other forms, or fell outside the scope of this round of testing.

While in many cases users expressed appreciation of CAC and considerable interest in its mission, this report focuses on areas in need of improvement. I hope you find it to be constructive.

    To help prioritize issues, I've included a “🚨” next to issues that I believe are of critical importance.

    Goal 1: Understand CAC's Purpose & Unique Value

    🚨Issue: The home page doesn't clearly state CAC's value proposition.

    A value proposition is a short summary of the outcomes the organization provides and what makes it unique.

    CAC's home page focuses on current events. It assumes visitors are already familiar with the center. 

    A few people in our testing did grasp that CAC is an independent, nonprofit cinema with films and special events that can't be found elsewhere. However, many did not. Several mistook it for a dance or theater organization — or a school of some kind. Others simply couldn't tell and were frustrated with that lack of clarity.

    For example, one woman who seemed to get the gist of what the organization provided nevertheless began describing CAC as a "club", presumably because of the reference to the premier club in the site's sidebar.

    This illustrates how important it is to be exceedingly direct and simple in directing visitors. Without a clear statement of purpose and value, visitors will cobble together inaccurate stories based on whatever clues they can find.

    Recommendation: Put a short value statement on the homepage, supported with comments from actual customers.

    Having a value proposition on the home page isn't just for those who are new to CAC. For returning visitors, a compelling statement can be a good reminder of why they should support CAC through attendance or membership. 

    Some people in our testing had an "aha!" moment when visiting the mission/history page, but almost no one read past the first, short paragraph. The value proposition on the home page shouldn't be a full account of CAC's history. A short statement, perhaps accompanied by a 60-second video would provide a couple of ways for people to quickly understand what makes CAC special.

    Now, you may be thinking, "If people want to know more about us, they can just visit the mission page."

    That's true, but some people won't bother to do that

    If the goal of the CAC website is to facilitate the sale of tickets and increase membership and donations (or "conversions"), then it's important to find every opportunity to reduce the cognitive burden on the user. Every single additional step will reduce website conversions.

    Moreover, a value proposition is going to make overtures for membership drives more compelling.

    Note: A value proposition isn't a statement of what the organization does. It's a picture of what outcomes the organization provides for visitors and supports. This could be focused on benefits of a community of film lovers who can enjoy films that they would otherwise have to drive for miles to get to, or wait until they become available on video. If you have customer interviews that relay statements about what actual people say they love about CAC, it's best to borrow from their vocabulary, as this will connect better with the reader.

    Untitled 4.jpeg

    While some users expressed appreciation of how much information the CAC website provides, they often felt the layout of the pages, the organization of the content, and delivery of information could be improved.

    Consequently, people often gave up before completing tasks, or they came to incorrect conclusions.

    Goal 2: Navigate the website with a clear understanding of how information is organized

    🚨Issue: A significant portion of users struggle to find information and find the presentation of the content lacking.

    We noticed people using words like "overwhelming", "it's just too much", and "confusing" to describe how they viewed the options to navigate the site and view content on specific pages, like the calendar, mission/history, and donation pages.

    Recommendations: Consolidate navigation and improve filtering

    Users were confused by the distinction between the calendar and the events pages.

    The Calendar and top-level Events links currently go to the same page, and the drop down menu items under Events often turn up a blank calendar.

    Create a single calendar page, perhaps labeled "Events & Showtimes". (The safest way to ensure good labeling is through further testing. I can share recommended techniques with you, if you like.) 

    Then, make the filtering function on this newly merged calendar/events page more prominent through size, color, and/or positioning, as some user seem to overlook the feature currently. 

    The user who discovered the calendar filtering function in our testing found it confusing. The hierarchy of options was unintuitive, and the dropdown filter required backtracking once special event options were selected:

    CAC might introduce simplified filtering based on a few essential categories, allowing users to choose from:

    • All Movie Showtimes & Special Events (default)
    • Movie Showtimes
    • Special Events

    When users select one of these options be sure both of the remaining options are still visible and can be selected without having to back up a step.

    Currently, when visitors want to view upcoming dates for a particular series of events, like The Classics or Cinema for Kids, they are met with a blank calendar. 

    Consider creating web pages for these events. On these pages, CAC could provide visitors with event descriptions, perhaps including images, of what these series are about along with any upcoming dates. 

    This would at least help people understand the types of events without being met with a blank calendar page, as is now the case with many events. Descriptive web pages should also increase traffic to the site from related searches in Google, which the site likely isn't getting under the current model.

    Issue: Users are confused by secondary navigation.

    fullsizeoutput_f513.jpeg

    Is the secondary navigation necessary?

    Check to see if people are using these links and, if so, how they're using them. Are they providing any benefit to users? How does behavior change when this element is altered or removed?

    Some users felt that the secondary navigation on the site was redundant, and some cited it as an element contributing to their feeling "overwhelmed". Redundant navigation items can create additional cognitive burden for users, as they revisit pages, wondering if they were missing important information.

    Recommendation: Run heat maps on the site and a/b test user behavior with and without secondary navigation.

    Heat maps will let you see where people are clicking on the site. Are visitors using the secondary links to complete important tasks? Test the site without the secondary navigation. How does that impact user behavior and revenue? I suspect the secondary navigation should be removed.

    fullsizeoutput_f515.jpeg

    Bug Alert

    When the website is viewed on a narrow screen, the navigation collapses into a mobile menu, which is expected behavior. However, on narrower screen widths on a desktop computer, the navigation items can disappear entirely.

    Issue: Social media icons distract from revenue-generating tasks and contribute to users' option paralysis.

    Several users identified the many social media icons as distracting and unnecessary. They asked why the icons were at the top and bottom of the site, and suggested they were contributing to their feeling that there were too many options to choose from on the home page. One user pointed out that they're almost as large as some of the home page images.

    Recommendation: Make social media icons less prominent, and bring them to the foreground only at those points in the customer's journey when it is most beneficial to CAC to do so.

    Acquiring social media followers can be important, but I would especially recommend CAC remove the icons from the top of the website. Social media engagement is a secondary goal for most organizations. Visitors who land on your site would be better served if they were first introduced to who you are and were then encouraged to complete a top task, such as purchasing a ticket or becoming a member. Neither of those activities can be completed on Instagram.

    Find ways to encourage social media activity after visitors have complete revenue-generating tasks, not before. For example, you might include social media links in confirmation emails or on confirmation pages post-checkout.

    🚨Issue: Membership and donation benefits are unclear to users.

    When users click on Make a Donation, some expect to see benefits listed alongside contribution levels — "like a kickstarter," as one user put it. Are donors also automatically members at a certain level? Some users weren't clear on this. 

    Recommendation: Test membership and ticketing flows for new and returning users.

    In our testing, users who did take initial steps toward donating expressed considerable frustration. Without more information, I can't make thorough recommendations, but my sense is that, while testing other goals, we've passed the tip of an iceberg here and there are major opportunities for improvement in this area that will positively impact CAC's revenue.

    🚨Issue: Users feel overwhelmed by the website's design.

    I've touched on this before, but want to introduce a design principle that CAC can use to address this issue in many of it's manifestations:

    Recommendation: Find opportunities to implement progressive disclosure.

    Progressive disclosure defers secondary or less common options for users to another screen or mode.

    For example, rather than including showtimes on the calendar page, CAC might show only thumbnails and titles. Then, when users click on the film they're interested in, the showtimes for that film on that day are shown. For the most part, users don't need to see showtimes until they know what movie and day they want to visit the cinema. Deferring times until they click reduces the amount of information displayed and can make it easier for users to come to a decision.

    Another opportunity to implement progressive disclosure is on the donation system.

    First, I want to point out that it's worth studying the entire donation process. Examine the sequence from a high level and place the most important and commonly used questions first. Optional information or less commonly used options, such as donating on behalf of a loved one, would be better placed toward the end of the sequence. Placing questions that some users will perceive as non-essential and superfluous early in the sequence increases abandonment, which means fewer donations.

    Once the sequence is optimized, look for ways to simplify further. For example, this page, which asks about restricted giving and giving on behalf of a loved one:

    fullsizeoutput_f516.jpeg

    Our testing shows that some users find this donation sequence frustrating.

    As one user said upon seeing this screen, "I want to give you my money and — done. Why can't I just give you my money?" Placing streamlined versions of questions like these these later in the donation process will increase conversions.

    Instead of asking people to fill in the blanks, ask "Do you wish to make a donation in memory of a loved one?"

    Provide them with a multiple choice option: Yes or No. 

    If someone chooses yes, then show them the options above. If they select no, skip those options. Conditional logic is a form of progressive disclosure.

    Next, you might ask, "Would you like your donation to be used for a particular purpose?" Then, let users indicate a "restricted purpose", using multiple choice options, such as "Soundproofing theaters one and two" or "Purchasing new screens". Also include an "other" option where people can write in their own purpose. This gives you a chance to gently guide donors toward particular purposes, while still allowing them to voice their own preferences. 

    I have a number of recommendations about the donation and ticketing systems. For now, I'll just emphasize that, while it may seem that implementing progressive disclosure and similar techniques are minor, I've yet to see a situation where changes like these don't markedly improve conversion rates (e.g., donations).

    fullsizeoutput_f51b.jpeg

    Note: Left-align copy to improve readability

    One user commented that the center-aligned copy was "distracting", and she wished it were "cleaned up". This may seem like a small detail, but it's a persistent usability issue throughout the site. It's rare to find a book or magazine with center-aligned text — that's because centered text can be distracting and reduces readability.

    🚨Issue: Visitors are missing key pages or only finding them by accident

    Desktop users, which made up all but one of our users in this collection of tests, don't recognize top-level navigation items as links. 

    I noticed that desktop users hover over a dropdown menu and click links within the dropdown, but never the top level navigation item.

    desktop cac menu.png

    Desktop users click the first option in drop down menus.

    None of our tests showed a desktop user who clicked top-level items in dropdown menus, though they were sometimes seeking the information contained on those pages.

    Recommendation: Make the first item in the submenu link to the same page as the top-level navigation item. 

     

    Issue: Carousels try to convey more information, but rarely succeed.

    In our testing, the carousel on the home page left some visitors feeling as if they "must have missed something" as they tried to gather information about CAC. The carousel largely relays more current events info, which can be found in the calendar/events info below the carousel.

    fullsizeoutput_f520.jpeg

    Auto-rotating carousels present considerable usability challenges.

    Studies show that very few people actually click on carousel items, and of the few that do, 98% are on the first item. Carousels also inevitably move too quickly for some and too slowly for others, which creates low level stress for the visitor.

    Here's a website that demonstrates why it's best to avoid auto-rotating carousels

    Companies tend to use carousels for two main reasons:

    1. They can't decide what to feature in the most prominent place on their home page, so they use a carousel to avoid making hard choices about what's most important. (In terms of addressing users needs and increasing conversions, what's most important is usually the company's unique value proposition, as mentioned above‚ not a carousel of news items.) Moreover, there are always competing interests within an organization (as it should be), but rather than resolve matters internally, the company deploys a carousel to try to satisfy everyone, which tends to only create frustrated or disengaged users.
    2. People are afraid that without the rotating carousel, their homepage will look boring. I'd point out that there are many ways to enliven a web page without flashing images — for example, video, interactive feeds, and even old-fashioned, well-written copy can be good alternatives to a carousel. I've never encountered a user who visited a home page that didn't have a carousel and heard them say, "I wish there were a carousel."
    Recommendation: Test carousel behavior and alternatives.

    With all that in mind, CAC might try turning off the auto-rotation function on the carousel, for starters. Perhaps update the first image in the static (not-automatic) carousel more frequently with the number one item you'd like to promote that day. Users can still scroll through the carousel manually, if they wish.

    Then, install heat maps on the site and see how many people actually scroll through the carousel.

    And test alternatives — for example:

    • If the content in the carousel is really important, why not put it on the home page where everyone can really see it? Break it out of a carousel entirely. How do users interact with the content now?
    • Try an introductory video where the carousel is now. Video seems like a natural medium for CAC. Study the video analytics to measure use.

    Goal 3: Sponsor as a business owner

    Problem: Completion rates for this task were very poor. Most users did not know where to go to complete the task, and those who did wished for a more streamlined approach.

    Most users assumed the only way to support CAC on behalf of a business was to make a donation as an individual. A few recognized Sponsorship as the correct path, but usually only after some trial and error. Those who did make it through to Sponsorship bounced back and forth between the main sponsorship page and the Opportunities page for some time before realizing they had to email or call to complete the transaction. Both of those individuals wished for an online form to complete the transaction.

    Recommendation: Improve navigation, introduce online forms & reduce the number of PDFs
    cinema-arts-mobile-menu.png

    Note: CAC's mobile menu doesn't offer all menu options.

    For someone looking for sponsorship options on smaller screens, the information isn't readily available. It's best to provide users with the same options and access to information across all devices.

    It would be best to offer an online form as an option here. The entire transaction — from making a contribution to uploading graphics for presentation on the CAC site — could be handled in a single form. 

    Similarly, offer an online form for the rate schedule. The current PDF option only slows the transaction down, and I would be quite surprised if an online form system doesn't increase the number of submissions CAC receives. (I've written more about the problems with PDFs on the web.)

    To help business owners find this section more reliably, it would be best to do a more comprehensive study of the navigation.

    Conclusion: What we did not test and what could come next

    CAC's website is a vehicle for ticket sales, membership sign-ups and renewals, and donations. The website does provide other benefits, but ticketing, membership and donations are the top tasks for the organization and its users.

    We would have tested ticketing, membership flows, and donating through the site, but additional resources are required to test actual transactions (e.g., refunding users testing the system).

    I've also stopped short of making more complete recommendations — particularly in regard to navigation and information architecture — because more study would be necessary to make informed, responsible recommendations.

    I hope you've found this analysis helpful and useful. I'm available to answer any questions you have, and I'd appreciate your initial, quick feedback using the form below. Thanks!

    Kyle Bowen