Summon Usability Testing (2010)

Overview

The Libraries have been investigating ways to improve discovery and retrieval of library resources, particularly for undergraduate students who can find the complex array of databases and e-journals difficult to navigate. The "Summon" application, developed by Serials Solutions, attempts to address these issues by providing a unified search interface.

Summon simplifies the process of finding information in the Libraries. It searches through a massive collection of books, scholarly journals, newspaper articles, e-books, dissertations, conference proceedings, and numerous academic databases all at once.

Purpose

The purpose of this study was threefold. The study was designed to assess how well students use a beta version of Summon, to determine how well Summon supports patron needs, and to establish recommendations for improvements to its overall design, functionality, and integration with other library applications.

Method

In March 2010, the Libraries hired consultant Abe Crystal (MoreBetterLabs, Inc.) to conduct user research on Summon, with a focus on the strengths and weaknesses of its information retrieval capabilities and user interface. Tasks included locating a known item, performing a citation lookup, and performing both simple research (article search) and complex research (research for current class assignment).

Team and Partners

Angela Ballard, study coordinator
Susan Teague Rector, study coordinator
Emily Lynema, Summon Product Team
Josh Boyer, Summon Product Team
Babi Hammond, participant recruitment
Susan Pauley, participant recruitment
Abe Crystal, MoreBetterLabs, Inc.

Recruitment and Interview Procedures

The first step in participant recruitment involved establishing the target audiences that stakeholders wanted to be represented in these studies. From here, the process involved further narrowing down the ideal demographics criteria for participants. With assistance from the consulting firm, the next step was to create a standard list of screening questions to be asked of potential participants. Libraries staff members were assigned the responsibility of recruiting participants. Ten student participants from various backgrounds, academic interests, and information literacy skills were recruited for this research.

Results

The amount of detail entered for known-item article queries varied greatly among participants, with more detail generally leading to search failure. Most participants had difficulty navigating from the citation to the full text of articles.
Book search success also varied. Some participants easily located books, while some had difficulty, particularly if they entered non-title elements to the title field (such as "5th ed."). Typographical errors caused some problems as well; Summon does not seem particularly forgiving of such errors compared to Google. Finding the catalog record for a book often required using Summon's filters to eliminate book reviews and journal articles related to the book's title. Only a minority of students were sophisticated enough to navigate this process.
When using the search interface, participants found the "advanced" search link unclear. The form was difficult to read, and labels were often unexpected ("written by" vs. "author"). Summon also modifies the query string after the Advanced Search form is filled out, which can be confusing for novice users.
Participants generally seemed to understand the search "filters" and showed some excitement about them. However, the length and wording of the checkboxes caused the filter list to be difficult to scan. Most participants ignored this area.
Subject term lists were sometimes overwhelming because of their length and complexity.
The slider widget showing publication dates, though engaging, were of little interest.
The article preview feature was useful to participants, although it is difficult to discover and was regarded as too small.
The "Save Items" feature was very popular and was convenient and useful. However, most students expected to log in with their NC State ID (which isn't supported). It was also hard to discover.
Participants loved getting access to full text of articles online, but this tool presents a risk of tab/window during exploratory research. Many students opened 10+ tabs within a few minutes of starting their research.

The post-test interviews suggested that participants liked the integrated nature of Summon and found it fairly easy to use. They likened the appearance to "Google for the library". Some planned to use it again for research papers.

Suggestions for improvement included:

better results ranking
greater ease of getting the full text of articles
less confusion over the relevance of subject terms
increased size of the article preview window

Undergraduate students appeared to value Summon much more than graduate students. Undergraduates' nonverbal (tone, expressions) communication about Summon was markedly more enthusiastic than graduates', and they appeared more likely to use Summon in their future work.

Project Links

Email screening questions asked of potential interview participants who responded to initial contact email or recruitment flyer.
Script for follow-up email sent after participant responded to screening questions.
Script for final confirmation of test session appointment with participant.

Reports and Presentations

Final Summon User Research Report, prepared by MoreBetterLabs, Inc.

Last updated: April 2010