Vol. 15 : No. 3
Editors Note: This paper is an excellent example of how Universities can work with industry to serve their mutual needs.
Distance Learning Software Usefulness and Usability: User-Centered Issues in Practical Deployment
By Cindy Ruman and Jay Gillette 
Blackboard CourseInfo is a software package that seeks to fill a niche in the distance learning market by bringing together various applications. Applications required include communication systems and a means to store and access documents. The question is how useful and usable is the Blackboard CourseInfo software to the students and faculty. This paper defines the key elements that are used within an on-campus graduate class and gives the results of surveys of that class on how useful and usable Blackboard CourseInfo was in supplementing the educational experience.
This study examined the usability of Blackboard through three approaches: user group interviews, usability testing sessions, and an end-user survey. In laboratory usability session, at least eight of nine participants were able to complete all of the tasks, taking, on average one to two attempts. Participants voiced their concerns and opinions through follow-up questions and survey ratings.
The most common response made by the study participants focused on the categorization of utilities within each menu button. Many tasks were not in the place that seemed intuitive. The most popular feature was the grade checking ability. Mean rating from the survey on ease of use and satisfaction were 3.58 and 3.56 respectively.
According to the survey, the most useful and usable item was the ability to check grades. Yet, ironically this is one of the most problematic items for professors to implement within their class
Staff and faculty interviews showed a reasonable degree of satisfaction, yet adjustments still need to be made, and the comfort level for this new software deployment will need to be increased to fulfill the needs of the users.
The higher educational realm is expanding from the classrooms of colleges and universities into the homes of collegiate students. Yet this new era of education may place students at a disadvantage because of a lack of face-to-face interaction with the professor.
Anne B. Keating (1999, pp. ix-x) has a practical approach to working in Internet-aided instruction, in her Wired Professor guide. She says:
The rational for using the World Wide Web to deliver course material, as well as to promote classroom interaction, rests in the potential that this technology has for providing twenty-four-hour access to information from any computer connected to the Internet. . . . By the time you read these words, the Web and related online technologies will have been further enriched by capabilities that we can only guess at now. This is the challenge of the Internet—the moment you take it on you become a participant in its development.
Blackboard CourseInfo is a widely deployed software that attempts to fill this potential by creating a package to assist in distance education by providing course material access and communication links.
Blackboard CourseInfo was designed to assist the interaction of students by providing multiple communication possibilities. Two asynchronous features provided by Blackboard are email and discussion threads. The use of the discussion threads also helps in the posting of a question or comment and allowing others to respond to the posting. The progression of the discussion can be followed chronologically. Synchronous communication is provided through a virtual chat feature. This is time-constrained but offers instant feedback for the students that are participating in the chat room.
Blackboard offers features to assist faculty that use the software by providing sections to post announcements, assignments, documents, and maintains a grade book. These sections provide information about any announcements or assignments made in class, eliminating the restrictions of distance that would prevent the students from accessing this information. For example, the Documents section is where professors can post syllabi, lecture notes, and other documents that may be needed by students. Another feature is the Grade Book section where the professors are able to post students’ grades allowing them to find their current status in class in accordance with the points that they have achieved versus the points possible.
also is focused upon information that the students can provide
themselves. The students have the capability to host a web
page about themselves to allow other students within the class to
know about their fellow classmates. This simulates a
community that can be found in the face-to-face classroom. A
further feature that is controlled by the students is the Group
section. This allows a group of students to collaborate
together and assist each other in assignments or projects,
simulating other interaction that may be found in a normal
Study participants included faculty, technical support staff, and graduate students in the various phases of the study. We expected that all participants had some knowledge of the product from periodic use during (at least) the past three months. The following summarizes the numbers of participants in the parts of the study.
Interview - 6 participants
Usability Testing Sessions - 9 participants
Survey - 30 participants
For the interview phase, participants were chosen from three user group levels: student, faculty, support staff. Different levels of users give a broader perspective of impact on users. The chart below gives some characteristics of this group.
For the usability testing sessions, volunteers were sought from a graduate course that utilized Blackboard during the present term (MGT 500 - the instructor of this course allowed our group to contact students). Nine volunteers participated in the sessions. Research shows that small numbers of users are sufficient to uncover most usability problems and that subsequent test users find the same problems already discovered (Nielsen, 2000).
A survey was given in the above course during one of the class sessions. The following are demographics about these participants.
The study included interviews conducted in person or by phone to ask about the experiences the different user groups had with the Blackboard software.
During the usability testing sessions, evaluators observed users at the computer performing pre-defined tasks and then asked follow-up questions. The tasks selected were representative of the types of features common to student use or potential use. Participants were given a written task list/instruction sheet for use during the session (see Appendix D).
A survey was given to assess subject ratings of product features.
Interviews were conducted in the participants’ offices or in another classroom environment. The usability testing sessions were held in a computer lab. No video or audio recording was conducted. The survey was administered in the regular classroom.
Participants in the usability testing sessions accessed Blackboard CourseInfo from any web browser (typically Internet Explorer or Netscape). Microsoft Word and PowerPoint software must also be loaded on the computer if such software was used by the professor to create materials for Blackboard. In the absence of these programs, participants would need knowledge of conversion techniques for other programs. Participants in this study used PCs (Apple computers also work) with monitors, and used the keyboard and mouse for input devices.
A customized questionnaire was compiled and given to those present in the graduate class (MGT 500). This survey can be found in Appendix A. Usability testing sessions were hand recorded on observation sheets (see Appendix B).
An instruction sheet for the evaluators was prepared so that all evaluators would be consistent in the way they conducted the sessions (see Appendix C). Participants were tested four in a group in a computer lab with one evaluator available to observe each subject. No interruptions were encountered in the use of this lab. Participants were greeted, the session purpose was explained, and their voluntary participation was acknowledged which included the right to leave the session at any time.
The evaluators then explained that subject would be given a list of tasks to complete, that the evaluator would be making notes of the session but was not judging the person. It was explained that the evaluators would not answer questions or assist in task completion but would intervene if there were a computer problem that prevented subjects from continuing. Participants were encouraged to “think aloud” and were given the opportunity to ask questions before the session began.
Evaluators marked task completion, number of attempts made to complete each task, and verbal comments made during the session. When participants concluded, follow up questions were asked and the participants thanked for their time.
The instruction sheet given to participants detailed login procedures, reminded them to think aloud, and enumerated each task they were asked to perform. Final items included logging out, answering questions, and a thank you for their participation (see Appendix D).
The evaluation team endeavored to assess the usability, usefulness, accessibility, and satisfaction encountered by participants using Blackboard CourseInfo.
This metric assesses the extent to which users can interact with the interface to complete a task. We marked whether or not tasks were completed to obtain a percentage of achievement. Similarly, the number of attempts was tallied to give an indication of the effort involved in understanding the operation of the interface to accomplish a goal. Along with these tallies, participants’ “think aloud” comments were recorded to gain insight into their thought processes as they worked through tasks.
The extent to which a user is satisfied with a product could impact future use and productivity. Subjective responses were obtained in all phases of the study. These include comments during the interviews and from the usability testing sessions, and ratings on the survey administered to the graduate class. The dimensions of satisfaction, usefulness, and ease of use were included in the survey, both in general questions and those relating to particular features. The survey utilized a 1-5 Likert scale with 5 meaning high or positive.
Student A found the menu items on the navigation bar to be vague. She expressed the desire to have a “pop-out” menu appear when the mouse points to each item. She also felt a more developed group area would be helpful. She did find the email and discussion boards to be beneficial features. She felt the weekly announcements could be very beneficial if the professor utilized them, or if they could link to the syllabus and automatically list that week’s readings and topics. Although she does not view Blackboard as a truly necessary tool to be successful in this class, she finds it helpful to have access to the Power Point slides.
One student in our interview group, Student B, mentioned he heard about Blackboard for first time in ICS 602:Human Communication (where it was available as an option), but never used it. He used it first time in this semester in MGT 500. He usually uses Blackboard once to twice a week to download the class handouts, to read assignments and announcements, and to check his grade. He said he has never had any problems logging on the program. It is easy to access. However, one problem he experienced is that there are too many sub-options under each button of the menu selection on the left-hand side of the main page. That is confusing.
Also, the system should show those sub-options on the main screen telling which is under which. So, the users do not have to do multiple steps hoping around between pages. He said, “If you don’t know the system, you have to look through in every button to find what you are looking for.” He suggested to do a better menu selections that shows sub-menus or the menu hierarchy that can be seen on the main page by rolling mouse without going to another page of every option. In overall he said “functionality” is good. The software assists him to complete his jobs. Also, he likes the “weather report” feature showing on the main page. It’s cool!
Interviews with Staff Member A of University Computing Services, and Staff Member B of the Center for Teaching Technology yielded many things. First, it afforded a perspective of Blackboard through the eyes of the support staff for Blackboard.
Some of the main problems that Staff Member B heard about from the users was both faculty and students seemed to forget their passwords quite often, and as a result they have to call for technical support. Staff Member B suggested possibly having an online drop box or complaint box.
We interviewed two professors on campus to gain a faculty perspective on the usefulness of CourseInfo, the pitfalls, as well as needed improvements or innovations in the implementation and use of the Blackboard CourseInfo software. For confidentiality, the professors here will be referred to as Professor A and Professor B. This was Professor A’s second semester using the CourseInfo software, whereas the use of the software was a new experience to Professor B.
Professor A and B mentioned some key benefits of the use of the software. Both discussed items such as the use of the bulletin board and posting abilities. They found that discussion could and would occur through this medium as well as it being a good place to disseminate information. Professor A utilizes the online assignment submission feature heavily as a way to grade and return papers quicker. Professor B liked the ability to post slides from class lectures as well as the use of the grade book system. Professor A, however, heard “horror stories” about the Grade Book and prefers to maintain the current system.
Both professors agreed that a considerable amount of time was required to prepare for a class, yet Blackboard focused their planning and organization. The extra work seemed to be appreciated by the students. The professors mentioned that their classes enjoyed being able to access the CourseInfo site and gain information about their course.
As with any implementation and use, concerns do arise. There seemed to be several issues that both professors voiced opinions about. First, simply managing the courses seemed to be a struggle. The implementation and initial setup seemed to be acceptable or bearable, but the daily maintenance or management of student accounts became troublesome for them. Secondly, in a scenario given, if a professor wished to send an email to all of the students, concerns arose as to the difficulties and time it took to change each student’s email address if he or she did not use the default Ball State email system. And for those who did use the Ball State email account, confusion arose between those still using the VAX system or those on the newer Microsoft Outlook system. This situation gave rise to multiple possibilities and difficulties in assuring that all students had updated and accurate email addresses for the course.
Third, Professor A mentioned that this was felt especially by non-traditional students who found it difficult to come to campus during office hours to acquire the needed VAX account and then access the appropriate people if questions or difficulties arose. Fourth, according to the professors, the sharing of written documentation and work should be much easier. Often, students use different word processing programs, making it difficult to open and read assignments sent to them. Fifth, a concern exists that there is no virus detection system or spell check for composing email or bulletin messages. And finally navigation seemed to be a point of difficulty for the professors. It takes many more steps, and waiting between clicks, to do something in Blackboard compared to the old way of doing things. Professor B mentioned that a nice feature to implement might be to add voice recognition.
This need for help in navigation appeared both in trying to find the correct place to go within the control panel to make changes as well as gaining enough access to support and training from the Center for Teaching Technology. Each professor gave ideas pointing to having written documentation, a question hotline, a student-focused help area, or somehow implementing a better-structured support and training system. However, both professors see value in the software, are committed to continue with the use of Blackboard CourseInfo in their future classes, and are optimistic about how the software could facilitate the educational process.
The following table presents the results of the usability testing sessions. The collective number of students completing each task, the mean number of attempts to do so, and any problems encounters or comments expressed during the test are shown.
The following are comments made in response to the follow-up questions after the testing sessions.
What parts of the interface were confusing, unclear, or hard to use?
What would you like to change?
What did you like about the interface?
Any other questions/comments?
Means and standard deviations were calculated on all survey questions. The following table lists the results. If respondents had not used a feature, they were instructed to leave the question blank. Therefore, it was not always equal to 30 and is included in the last column.
Additionally, we asked for ratings on usefulness and usability of the features in particular.
The highest ratings (3.7 or above) were given to encouragement of instructor, ease of access over web, discussion threads, checking grades, submitting assignments online, announcements feature, beneficial that software brings utilities together, and desire for use in future classes. Lowest ratings (below 3.00) were provided for the group and personal web page features, and virtual chat. These were also the features receiving the lowest response rate: only half the respondents even marked a rating.
Correlations were obtained between some of the variables on which we especially wanted to focus. These correlations meet or exceed a p < .002 level of significance.
There were no significant findings that previous experience with the product impacted ratings of satisfaction, ease of use, or desire for future use.
The following graphs illustrate the frequency of each rating level chosen by the respondents on selected dimensions of the survey (5 = high or positive).
Some of the features in Blackboard are seen as more usable or useful than others. The chart below lists subjective ratings obtained from survey responses. Applicability of certain features is somewhat dependent on their required or suggested use in any given course. The instructions in the survey asked respondents to leave an item blank if they did not use the feature, thus, those that chose a rating had an opinion to express.
It is interesting to see the mean averages from the survey results on the various options that Blackboard provides. According to the survey, the most useful and usable item was the ability to check grades. Yet, ironically this is one of the most problematic items for professors to implement within their class. Other helpful features include the announcements section and the ability to submit assignments online. These features facilitate gaining information to complete tasks that the professor requests. The students are then able to provide the completed assignment back to the professor for a grade. This provides complete functionality of a class outside the physical classroom: just what Blackboard was looking for in providing a distance education software package.
The ratings of the students were above average on how satisfied they were with Blackboard, if they would use it again, and ease of using the interface. This is promising for the software, but does not display any outstanding benefits for Blackboard either. Even the most basic issues of password access caused some difficulty for students.
The lowest rated item for usefulness was the personal web page and the lowest rated item for usability was the group web page. This is most likely due to the lack of perceived need to learn to use this particular feature in the present course. Our view is that the usefulness and usability of these options may increase with use by distance education groups wanting to interact with their team members. However, these functions could still be replaced by personal web pages and email communications from the group members using other, more familiar and less expensive software packages. (More information on the mean values of the survey can be found in Appendix A.)
Participants most often commented on how confusing it was to determine which main category (left hand menu) a particular task fell under. They wanted to see pop-up menus when they moved the mouse over the item.
Much of Blackboard’s potential success has to do with the extent to which professors incorporate it into their classes. Blackboard has potential for considerable functionality. It’s up to the circumstances of a class to make use of it.
Blackboard User Survey for MGT 500: Spring, 2000
The following questions ask about your experiences with using Blackboard CourseInfo. Please fill in your answers or use the 1-5 scale to circle your rating (with 5 meaning High or Positive).
(If you have not used a feature that is discussed within the question, please leave it blank)
Age: 28 Sex. M 20 F 10 Part-time 6 Full-time 24
1. Have you used Blackboard CourseInfo in a previous class? Yes 16 No 14
many times per week, on average, did you access Blackboard
3. How encouraging was your instructor to use Blackboard? 3.8 1 2 3 4 5
4. Please rate the ease of accessing the Blackboard software over the web browser?
3.9 1 2 3 4 5
5. Have you experienced any difficulties in:
Password access Yes __21___ No ___9___
Browser Yes ___2___ No __27___
Accessing URL Yes ___9___ No __20___
Error messages Yes __11___ No __18___
Other Yes ___5___ No __19___
6. Please rate both
how Useful and Usable you find the following
Observation Record Sheet for Usability Testing
Blackboard CourseInfo Project
April 13, 2000
What parts of the interface were confusing, unclear, or hard to use?
What would you like to change?
What did you like about the interface?
Any other questions/comments?
Instructions for Evaluators
The Blackboard Project Team is studying how users interact with the Blackboard CourseInfo software interface. We want to see what issues may be encountered. Software designers can benefit greatly from this information, and the company can improve its product.
Explain Voluntary Participation
Thank you for volunteering to help us evaluate this product. You are free to leave the session at any time if you wish.
Explain Your Role and Tasks
We will give you an instruction sheet explaining each task you will be asked to do. While you are performing the tasks, we will be making notes on the process. Be assured that we are not judging your performance, but are looking at how users interact with the interface. Our role will be to observe with minimal interaction, and thus will not tell you how to do a task. However, if a circumstance arises that prevents you from being able to continue, we will assist.
Explain That Participants Should Think Aloud
Part of the benefits we gain will come from your comments as you work through the exercises. You are strongly encouraged to think aloud as you go and verbally explain what you’re doing, what questions come to mind, and express any frustrations or satisfactions you feel.
Do you have any questions before we start?
Give Instruction Sheet - Mark the Observation Record Sheet as appropriate
At the Conclusion
(Ask follow-up questions listed on the recording sheet.)
Do you have any additional comments or questions?
Thank you for your
time and efforts.