Introduction 

The Role of Theory
in HCI


Development Methods

 Ethnographic 
 Methods 


Controlled
Experimentation


Survey Methods

Logging &
Automated Metrics



CHARM
Choosing Human-Computer Interaction (HCI) Appropriate Research Methods




Development Methods: User Needs Assessment & Task Analyses

A. Sasha Giacoppo (asgiacoppo@yahoo.com)
Department of Psychology
Catholic University
Washington, DC 20064 USA
October 2001

1. Introduction

Understanding who users are and what they are doing can and should be a critical component of your HCI investigations. The techniques and methods used to obtain user and task information come from usability engineering, which is a group of development methodologies that combine engineering, development, and usability methodologies. Utilizing these techniques and methods in your HCI research can support your investigations, or help identify areas for more in-depth investigations.

Logical User-Centered Interaction Design (LUCID) (www.cognetics.com/lucid) is an example of a development methodology that infuses the typical engineering and development process with usability and research methodologies. The boundaries of development and research methods are not exceptionally clear, they are intended to complement each other. Other examples of development methodologies, which can be found at www.otal.umd.edu/guse/methodologies.html, are:

  • Measuring Usability of Systems in Context (MUSiC)
  • Contextual Design Performance Centered Design
  • Scenario-Based Engineering Process (SEP)

The research method components of usability engineering differ from more purely scientific endeavors, in that they are focused on provided qualitative and quantitative information about users. They are used to understand the user, the user’s interaction with a tool/technology, the tasks that the user performs, and the social and environmental influences on users and their tasks.

The primary research method used in development is a User Needs Assessment/Task Analysis. The major tools of this method are:

  • Task Analysis
  • Contextual Inquiry
  • Verbal Protocols
  • Interviews

The methods available to researchers and designers for a user needs assessment and task analysis range from more formal methodologies that intend to collect quantitative data to more informal, observational methodologies that collect qualitative data.

As common component of scientific research within Human-Computer Interaction (HCI) involves a preliminary understanding of your users and the tasks that they perform to achieve their goals. According to Hackos (1998), a user needs assessment and task analysis seeks to understand:

  • What user’s goals are; What they are trying to achieve
  • What users actually do to achieve those goals
  • What personal, social, and cultural characteristics the users bring to the tasks
  • How users are influenced by their physical environment
  • How users’ previous knowledge and experience influence how they think about their work and the workflow they follow to perform their tasks
  • What users value most that will make a new interface be a delight for them (i.e. speed? Accuracy? Error recovery? Human contact? Fun?)

A user needs assessment and task analysis is perfectly suited for development-oriented researchers and designers. The results of user and task analyses can be used in several phases of the development process (Tucker, 1997):

  • Development of requirements
  • User Interface Design & Evaluation
  • Follow-up After Installation/Use of System/Product

It provides a broad understanding of users, their needs, and the users’ environment. This information has great utility for researchers by identifying potential areas for further, in-depth investigation.

2. Methodologies

2.1. Task Analysis

A task analysis is not a single analysis, but rather a collection of many different analyses. A researcher must decide which analyses are needed, based on the amount of resources available, time, etc.

2.1.1. Techniques of Task Analysis

Workflow analysis (business process analysis)

Description: Performed to understand how a user, or a group of users, accomplishes a particular process. Also known as “business process analysis”, workflow analysis is a horizontal picture of how work moves across and between people. Workflow analysis is important because it allows investigators to identify which users are doing which tasks in a process.

Importance to HCI: Though business process analysis is not necessarily a HCI specific research method, it does support HCI researchers in their understanding of users. More importantly, workflow analysis provides HCI researchers with important information of user behavior in context.

Job analysis

Description: Performed to understand the activities of a single user within a process. Job analysis is a vertical picture of all the activities that flow through a single user. A job analysis should involve direct observation of a person over time, or at the very least, a logging of daily activities by a person.

Importance to HCI: Job analysis can capture meaningful, quantitative data about how and where users spend their time. Information like frequency, time to complete, difficulty, etc. can all greatly help researchers designers and developers when it comes to developing, improving, discovering, etc. interfaces and products for users.

Combined Workflow/Job Analysis

Description: Combining the individual workflow analysis with the individual job analysis. The synthesis requires significantly more effort and coordination than the individual analyses.

Importance to HCI: A workflow and job analysis in tandem can provide researchers with a broad and deep picture of the users involved in a process, their tasks, and their interactions. This information assists developers and designers make educated decisions about the types of tasks and functions that are needed in products.

Task lists and inventories

Description: A task list is an inventory of all the tasks that users will want to accomplish with a tool, and sometimes includes additional tasks the users may want to have in the future. It is a list of what users have to be able to accomplish, not how to accomplish those tasks.

Importance to HCI: A task list will help drive decisions on the product’s functionality.

Process analysis,task sequences

Description: A task sequence is a series of tasks on the lists that users must do, or are likely to do in a certain order (e.g. a user must write an email before sending it). A task sequence of users is an investigation, optimally through natural observation. It provides a picture of the steps users take to go through tasks.

Importance to HCI: Process analysis and task sequences can be used to identify areas within interfaces/tools that are inefficient, error-prone, missing, etc. It is yet another technique that help researchers understand and improve designs.

Task hierarchies

Description: A task analysis is hierarchical. A task hierarchy illustrates the tasks and subtasks of a process or activity. It seeks to deconstruct and decompose tasks into its smallest components

Importance to HCI: It provides researchers with a picture of how many levels of tasks are involved in a process. The researcher can then decide which levels and how many levels of a process need to be investigated.

Procedural analysis

Description: One specific task that is divided into the steps and decisions that a user goes through in doing that task. It illustrates how users carry out their tasks with the tools currently available to them.

Importance to HCI: Procedural analysis can create an idealized picture of the process that users follow, or an actual picture of the process that users follow. From this, areas that need to be improved, or where users make a lot of errors, can be identified, or investigated further.

2.1.2 Advantages/Disadvantages

Advantages: Task analysis is a complex battery of techniques intended to provide a researcher with a complete understanding of what tasks a user really performs, what is needed to complete those tasks, and what tasks a user should be doing. A task analysis, depending on the depth of your investigation, provides a multi-faceted picture of a task/group of tasks that can help you drive investigations and research regarding HCI issues.

Disadvantages: A task analysis can be an extremely time-and resource-consuming affair. The more information you desire to collect, the longer you will have to burden users with observation and data collection.

2.2 Contextual Inquiries

Contextual inquiries are observational techniques that collect data by observing the activities and behaviors of users. Observation of users ranges from viewing/recording users in their typical environment to viewing/recording users in atypical environments. Contextual inquiries help produce information not only about user behavior, and also elucidate the impact of the user’s social interactive and cultural environment on behavior. The Infopolis 2 Consortium website(see references) provides an excellent description of the advantages, disadvantages, and descriptions of contextual inquiry methods and techniques, listed below.

2.2.1. Advantages/Disadvantages

Advantages: Observational techniques produce information which cannot be collected by any other means. It provides a clear picture of user behavior, but more importantly, it can provide a picture of users in their natural environment. Exploratory observational studies allow the investigator to familiarize themselves with the tasks that users actually perform, rather than the simple outcomes. Observation studies can be used to identify and develop explanations of individual differences in task performance. Observational techniques provide data which can be compared and contrasted with information collected by another investigator, or by another method

Disadvantages: Observation is a time- and effort- intensive activity; observational data contains an extremely wide amount of information. Direct observation can be an intrusive technique. The presence of an observer may also influence user behavior. A researcher must be aware of how much of an impact the selected type of observation will have on the user. The equipment needed for producing high quality observational data can be expensive and difficult to install, and video recording followed by later analysis is time-consuming. Proper observation demands properly trained professionals.

2.2.2. There are three levels of observation available to researchers:

Passive:

Passive observation is where the person conducting the observation quietly observes and records the user activity. Passive observation has the advantage that it minimizes the intrusion which the user experiences. One problem with passive observation is the practical difficulty in recording sufficient detail of the activity. This can be overcome through the video recording of user activity which, after transcription into textual form, can provide a highly detailed analysis.

Participant:

The investigator takes part in the tasks alongside the users. It may be a useful data collection approach if skilled performance is such that actions are "semi-automatic" in which case users may find it difficult to verbalize how they are achieving the task goals. This type of observation can also be useful if aspects of team performance are being investigated to understand how the team members are organized and perform their tasks.

Communication/Conversational:

This is type of observation is an investigation of the patterns in the organization of people's interaction. One of these methods is the language / action approach which considers the language as a means by which people act. The advantage of this method is that it provides a complete and logical conceptual frame in order to investigate all kinds of conversation. The disadvantage of this method is that in many situations are characterized by subtle communication processes not taken into account by the model. There is also a difficulty in labeling an interaction and a message especially if they do not fit into the request or promise categories.

Links:

http://www.best.com/~jthom/usability/context.htm

A simple who, what, where of contextual inquiries, with linked references

2.3 Verbal protocols (aka Cognitive Walkthrough)

A technique that asks users to describe their personal actions and thoughts while performing a task. The Infopolis 2 Consortium website(see references) provides an excellent description of the advantages, disadvantages, and descriptions of the verbal protocol.

2.3.1 Advantages/Disadvantages

Advantages: Verbal protocols are useful because they contain the user’s spoken observations, which provides insight into the cognitive processes of the user while performing a task. The information retrieved from a verbal protocol will be wide, and can highlight a multitude of issues regarding a task/interface

Disadvantages: Verbal protocols place an added strain on a user during a task, which may affect the user’s ability to perform a task

Links:

http://www.cc.gatech.edu/computing/classes/cs3302/documents/cog.walk.html

A description and Q&A of cognitive walkthroughs.

2.4 Interviews

Interview techniques are useful for identifying possible areas for more detailed analysis. Interviews are easy to conduct and direct, the unstructured interview can generate interesting points, statistical analysis can be run on the users answers. The data collected provides information about general rules and principles and is faster than observational techniques. Interview techniques are useful for investigating events which occur infrequently. The interviews can be recorded for a future analysis. The Infopolis 2 Consortium website provides an excellent description of the advantages, disadvantages, and descriptions of interviews and interview techniques.

2.4.1. Advantages/Disadvantages

Advantages: Interview techniques are useful for identifying possible areas for more detailed analysis. Interviews are easy to conduct and direct, the unstructured interview can generate interesting points, statistical analysis can be run on the users answers. The data collected provides information about general rules and principles and is faster than observational techniques. Interview techniques are useful for investigating events which occur infrequently. The interviews can be recorded for a future analysis.

Disadvantages: Respondents are not committed to give correct answers and may often be influenced by what they believe the interviewer requires, or what they themselves wish to portray. The interviewer may need to acquire domain knowledge in order to know what questions to ask.There is a range of considerable bias due to the understanding by the users of the questions, and the subjective collected information might be misleading or inaccurate. The critical aspects are the choice of the place for the interview and how to conduct it.

2.4.2. There are three types of interviews:

Unstructured:

An unstructured interview is when the interviewer does not impose any controls on the interview or the respondent. It is a intended to act as an exploratory conversation. Unstructured interviews are good for investigating potential emotional and / or sensitive personal issues.

Semi-structured:

Semi-structured interviews should only be carried out in a situation where broad issues may be understood, but the range of respondents' reactions to these issues is not known or suspected to be incomplete. This type of interview is mostly applicable in situations where both qualitative and quantitative feedback are required.

Structured:

Structured interviews are useful in situations where the respondents range of replies may be estimated and there is a need to clarify details, opinions or ideas. Structured interviews work well when the assessment goals are clear.

3. Recommendations

A user needs assessment/task analysis helps a researcher ascertain what tasks a user performs, and what tasks a user needs to perform. A task analysis can be a deep and broad investigation of those needs, albeit, at a considerable expense of time and cost. However, selecting one or two assessments of task analysis can be done at a lower cost, while still capturing valuable information.

The capturing of data means first deciding on the level of invasion and interaction the researcher wants to do, or is allowed to do. Next, the methodology needs to be selected, based on the level of interaction that has been decided upon. Contextual inquiries are inherently observational, but can involve interaction with the user, which can sometimes have an affect on behavior. Interview techniques are numerous as well. They provide more qualitative information and allow a user to express their opinions and information.

Whatever the case, a user needs assessment/task analysis will help drive your research, and identify potential areas of investigation.

3.1 References:

Hackos, J.T. & Redish, J.C. (1998) User and Task Analysis for Interface Design. John Wiley & Sons, Inc.

Infopolis 2 Consortium. From http://www.ul.ie/~infopolis/methods/interv.html

Keiras, David (1997). The Computer Science and Engineering Handbook. CRC Press, ACM.

4. Links:

http://usability.gov/methods/data_collection.html

This is the governments website for usability, which is usually less technical than other industry websites. This link provides descriptions and comparisons of the various methods used in collecting data from your users.

http://www.ul.ie/~infopolis/methods/interv.html

This website provides a list, comparison, and the advantages and disadvantages of several task-analysis methods.

http://home.netscape.com/computing/webbuilding/studio/feature19980729-2.html

Another website that is in “English”, and describes the various methods used in task analysis.

http://www-106.ibm.com/developerworks/library/moderator-guide/requirements.html

Put out by IBM’s research group, this website provides another excellent list of user needs/task analysis methodologies.

http://www.teced.com/ue-fs.html

Another site that provides a description of the various development and user needs/task analysis methodologies.