CS 486 – Capstone Project

Usability/Functional Testing Plan

Submitted to

Dr. Doerry

By

Team Fugu:

Erik Wilson

Ben Atkin

Nauman Qureshi

Thad Boyd

On

April 12, 2004

Introduction

Team Fufu is now finished with most of our major milestones for OpenBSD Tools for the USGS, and we have now designed and built an almost fully-functional piece of software. The testing phase is one of the most critical phases for our software project because it allows us to identify any major weaknesses which might exist. There are a number of possibilities for testing and evaluating our software, the tests and evaluations most applicable to our schedule and described in the sections of this document are Expert Reviews, Usability (Assessment) Tests, Acceptance Testing, and Analyzing the Results.

Expert Reviews

The software that we developed will require more than just basic knowledge of computers to operate it. Because installing and maintaining UNIX style operating systems are more in-depth than other operating systems, we require that our expert reviewers be more trained in the system administration aspects in order to correctly operate this software.

For the process of expert reviews we have selected the following three system administrators who are knowledgeable of UNIX:

Tom Baca

Scott Hanncock

John Huerta

The areas that the expert reviews will focus on are stated as follows:

Installation Configuration Files

Installation Process

Installation Logs

Patching Configuration Files

Patching Logs

These areas of focus will allow us to sollicit suggestions on the major functionality of our software for improvement. Any suggested improvements will be taken into consideration and implemented before Assesment Studies begin on April 14th. The next section describes in greater detail the process of Assesment Studies and how we will gauge the product's performance for the given target audience.

Usability: Assesment Studies

The main objective of testing your product before handing it over to the client is to ensure that our product is fully functional and bug-free. To avoid the mishap of an unusable product we have created three testing groups who would be conducting the assesment studies for the program. The following are the groups and dates of the studies:

Group 1: Tom Baca and Shanadeen Begay	(April 14th)

Group 2: John Heureta and Brian Adams	(April 15th)

Group 3: Scott Hancock and Jay Anderson	(April 16th)

The intention of these groups is to engage in constructive interaction without the aid of any designer or facilitator intervention. Each group will be allotted aproximately 15 minutes to perform their tasks. To best simulate realistic domain tasks the testing procedures below will be included in the lab manual and performed by each of the three groups:

Open in an editor the file called “floppy.conf” from the floppy drive.

Modify the configuration file under the “Disks” section to partition a disk with a 1 gigabyte swap, 500 megabyte /usr partition, and the remaining space allocated to the root partition.

Under the “Logging” section define the logging URL as http://www.cet.nau.edu/~fugu/log/logger.pl

Under the “Tepatche” section define the location for binary patching downloads as: 13.114.16.xxx

Under the “Tepatche” section define the appropriate key to use your email address as the location to send logging messages for the patching process.

Begin the OpenBSD installation process by inserting the floppy and CDROM, and then reboot the system.

From the logging website (http://www.cet.nau.edu/~fugu/log/logger.pl) monitor the progress of the install, when the install is finished this test has been completed.

In addition to the above procedures the lab manual will contain detailed information on the structure of the configuration file as well as examples. Email and web browsing clients will also be provided on a seperate computer other than the one OpenBSD is to be installed on. The next section describes Acceptance Tests of the newly installed software in the form of maintenace.

Acceptance Test

For our usability testing, we have set certain time criteria for the maintenance processes of the project. The maintenace of the newly installed operating system is critical to alleviating the time consuming tasks which exist in the current manual processes. This maintenance most often takes the form of determining if any errors occurred in the installation or patching processes. Performing these tests will allow us to consider our program’s user interface in the form of maintenance as accepted or rejected.

Using the logging website (http://www.cet.nau.edu/~fugu/log/logger.pl), record the number of errors which occurred during the install process, the severity of those errors, and the messages of those errors.

Check the tepatche log sent to the email address defined in step 5 of the assesment study, record the number of errors which occurred during the patching process, the severity of those errors, and the messages of those errors.

Each step should be completed within aproximately 30 seconds. Timing begins once the website or email client has fully loaded (i.e. after any passwords have been entered), and ends when the number of errors, severities, and messages have been recorded. Several emails or simulated errors may be included for testing purposes.

Analyzing the Results

First, we will analyze the expert review and summarize the problems identified; then put them in some priority order. For each, we will discuss and note issues like problem validity (i.e. do you think this is actually a problem), severity, and what we intend to do about it. Any problems that, at this point, seem obvious issues, we will fix them; others that are questionable we will examine in user testing.

The next analysis focuses on the assessment studies. Steps:

For each tape we will take careful notes on the nature of breakdowns, and what seems to be causing it.

We will identify patterns, i.e., group the low-level confusions into larger categories, e.g., anything from "problems with setting up partitions" to "don't understand the logging metaphor".

For each error we will develop one or more solutions to pursue to solve the problem. From this we can implement fixes quickly.

Repeat steps 2-4 for each of the three sessions in our analysis.

When we are done, we will create a summary of our assessment studies for presentation in your final report, and in the Capstone presentation: re-visit the problems found and say which ones were fixed, and which ones remain untouched, and what this means for our product's usability.

The last step is the acceptance test analysis, which is more numerical. We will go through the tapes, collect times data from the tapes. Then we will present data, statistics, and results on each of the performance specs you measured in the form of a graph.

CS 486 – Capstone Project

NAU Student

2004-04-11T21:55:00