Introduction 

The Role of Theory
in HCI


Development Methods

Ethnographic
Methods


Controlled
Experimentation


Survey Methods

Logging &
Automated Metrics



CHARM
Choosing Human-Computer Interaction (HCI) Appropriate Research Methods




Survey Methods: Questionnaires and Interviews

Ugur Kuter (ukuter@cs.umd.edu)
Cemal Yilmaz (cyilmaz@cs.umd.edu)

Department of Computer Science
University of Maryland
College Park , MD 20742 , USA

November 2, 2001

1 Introduction

Survey research is one of the most important areas of measurement in applied social research. The dictionary meaning of survey is: “Survey is a technique for gathering information from a large number of users” [Brehob, 2001]. A “survey” can be anything from a short paper-and-pencil feedback form to an intensive interview.

Surveys are an important technique used in Human-Computer Interaction (HCI) research. They provide feedback from the point of view of the users. They provide information regarding users’ preferences and ideas about the design in many stages of the interface development. Users’ reactions can have a strong impact on the design and development of an interface. However, not all surveys are useful. The data collected in a survey can be biased. This means that the answers to some kinds of questions - for instance, those related with time measurements or frequency of event occurrences - may not be reliable.

In general, the steps in designing and conducting a survey can be listed as follows [Trochim, 2000]:

  • Set the goals - What do you want to capture?
  • Decide on the target population and sample size - Who will you ask?
  • Determine the questions- What will you ask?
  • Pre-test the survey - Test the questions
  • Conduct the survey - Ask the questions
  • Analyze the data collected - Produce the report

The key step in designing a survey is setting the goals. The goals of the survey determine the target population and questions. If the goals are not clear, the result of the survey will be uncertain. Correctly determining the target population is critical; it should represent the targeted users of the interface and bias should be eliminated. This concept is known as sampling. Sampling is defined as “the act, process, or technique of selecting [...] a representative part of a population for the purpose of determining parameters or characteristics of the whole population” in Sampling In Research, which is a web tutorial on the subject by Mugo Fridah W.

In determining the questions, there are issues that must be considered such as type of questions, question contents, response format, question wordings, and question placing [Trochim, 2000]. The following paragraphs explain these issues briefly.

Researchers use three types of questions in surveys, namely multiple choice, numeric open-end (e.g., on the average, how much time do you spend per day on this system?) and text open-end (e.g. please give your comments about ...) [Trochim, 2000]. 

Mainly, there are two types of response format: structured response and unstructured response [Trochim, 2000]. Structured responses are very easy to be answered by the respondents but might not capture everything in the respondents' mind(s) (e.g. responses to multiple-choice questions). In unstructured responses, the respondents write down text as a response (e.g. responses to text open end questions). Questions should be clear and unambiguous. Also, the order of the questions matters. For example, the easier questions should be placed before the harder questions. The rationale behind this is to prevent respondent boredom at the beginning and to motivate them to complete the survey.

Researchers should prepare the questions and answer choices with a statistical method in mind, because they will be used to analyze the collected data at the end. However, not all the statistical methods might be applicable for a particular response type. For example, there is no statistical method that can analyze text open-end questions. There is a huge amount of literature regarding statistical data analysis. For example, a Summary of Survey Analysis Software is provided by Survey Research Methods Section (SRMS) of the American Statistical Association (ASA). Some useful links on statistical methods are “Statistical Data Analysis: Inferring from Data” and “Research Methods & Statistics Resources”.

Researchers should test the questions on a small sample of users, and they should also analyze the collected data with the planned statistical methods before the survey is conducted on a large scale. This gives the researcher a last chance to address possible problems. A survey can be created and applied at any step of a user interface design process. The rationale behind this is to identify the problems as early as possible to prevent researchers from wasting time and money [Hom, 1998].

While designing the survey, it is not sufficient to consider only these issues. Researchers should also be aware of the ethical issues in preparing, conducting, and analyzing the collected data from surveys. American Statistical Association provides “Ethical Guidelines for Statistical Practices.” These are general guidelines for researchers who do statistical analyses and/or conduct any type of experiment on human or animal subjects.

This report focuses on two forms of survey-research in detail: questionnaires and interviews. The following sections present the definitions, advantages and disadvantages of each survey-research method, as well as some commonly known examples from the literature and links to those examples. Finally, we conclude our discussion and provide some recommendations.

2 Survey Methods: Questionnaires

In usability glossary,  [Brehob, 2001] defines a  questionnaire to be "a form that people fill out, used to obtain demographic information and views and interests of those questioned". [Kirakowski, 1998] defines a questionnaire in a more structural way as "a method for the elicitation, and recording and collecting information". HCI researchers use questionnaires as tools (methods) to capture what is in users' mind(s) (elicitation). The data collected from a group of respondents (users) is recorded onto a permanent medium to be analyzed and referenced later.

There are two main types of questionnaires, namely mail questionnaires and on-line questionnaires. The latter can be divided further into three subgroups, i.e., e-mail questionnaires, computer-direct questionnaires, and Web-based questionnaires. This report investigates only the Web-based questionnaires.

As described in Survey Design, Questionnaire Design Tips by Creative Research Systems, mail questionnaires are sent to the respondents via surface mail. An important advantage of mail questionnaires is that they are inexpensive. They can also include pictures. Note that this is not possible with some survey methods e.g. phone interviews - which will be described in the next section. Another advantage is that mail questionnaires allow respondents to complete the questionnaire at a convenient time. This is not the case with phone or personal interviews as the respondents are obligated to the time slots provided by the interviewer.

Mail questionnaires also have some disadvantages. First, it may take several weeks to collect the responses, which may be too long. Second, the response rate is usually less than 5%. Finally, mail questionnaires could lead to a biased sample since, most of the time, researchers do not know the background of the respondents. To some extent, it is possible to collect information about the respondents' background by including related questions into the questionnaire.

Web-based questionnaires are rapidly gaining popularity as the Internet and World-Wide Web usage increases. Survey Design, Questionnaire Design Tips lists the advantages of web page surveys as follows: First of all, Web-based questionnaires are very fast. It is possible to get (a) large number of responses (e.g. several thousand responses). Secondly, the order of the questions (and even the number and type of questions) can be changed according to the respondents' answers. Finally, the questionnaire can use multimedia content (e.g. sounds, videos etc.), colors, fonts, and different formatting options.

Although the Internet is gaining popularity, it is still far from being universal. Therefore, Web-based questionnaires may not reflect the whole population. Also, Web-based questionnaires could lead to a biased sample because there is often no control over the respondents. People with a varying spectrum of background from all over the world can complete the questionnaire. Moreover, some of them may respond to the same questionnaire multiple times.

In general, questionnaires are designed to assess aspects of usability, the validity and/or reliability of human-computer interfaces [Pearlman, 1998]. In [Pearlman, 1998], some of the well-known questionnaires in HCI are listed as follows

  • Questionnaire for User Interface Satisfaction (QUIS), Univ. of Maryland , 1988
  • Perceived Usefulness and Ease of Use (PUEU), IBM, 1989
  • Nielsen's Attributes of Usability (NAU), Bellcore, 1993
  • Nielsen's Heuristic Evaluation (NHE), Bellcore, 1993
  • Computer System Usability Questionnaire (CSUQ), IBM, 1995
  • After Scenario Questionnaire (ASQ), IBM, 1995
  • Practical Heuristics for Usability Evaluation (PHUE), OSU, 1997
  • Purdue Usability Testing Questionnaire (PUTQ),  Purdue Univ. , 1997

2.1 Example: The Questionnaire for User Interaction Satisfaction (QUIS)

QUIS is a tool developed by the Human-Computer Interaction Lab (HCIL) at the University of Maryland at College Park . QUIS aims to assess users' subjective satisfaction with specific aspects of the human-computer interface.

QUIS is a measure of users' overall system satisfaction. It also contains eleven specific interface factors that are organized hierarchically, namely screen factors, terminology and system feedback, learning factors, system capabilities, technical manuals, on-line tutorials, multimedia, voice recognition, virtual environments, internet access, and software installation. Each factor measures users' satisfaction with the general properties of the interface as well as the specific ones.

The three questions below are taken from [Shneiderman, 1998] and they are related to screen factors in an interface design. Question 4.1 evaluates the user satisfaction with the general properties of screen characters whereas the others are about relatively specific ones.

    4.1 Characters on the computer screen            hard to read            easy to read
                                                                            1   2  3  4  5  6  7  8  9        N/A

        4.1.1 Image of characters                            fuzzy                       sharp
                                                                            1   2  3  4  5  6  7  8  9        N/A

        4.1.2 Character shapes (fonts)                     barely legible         very legible
                                                                            1   2  3  4  5  6  7  8  9        N/A

For more sample questions the interested readers should consult [Shneiderman, 1998] and the QUIS (http://www.lap.umd.edu./QUIS) web page.

2.2 Questionnaires CASE STUDY: GVU's WWW User Surveys

GVU's WWW User Surveys are designed at Graphic, Visualization, & Usability Center (GVU) at Georgia Institute of Technology. These surveys are a series of web-based questionnaires started in 1994. At the beginning, they aimed to investigate the trends in the Internet usage. Recently, the questionnaires are expanded to include commercial uses of the Web, including advertising, electronic commerce, intranet web usage, and business-to-business transactions.

Currently, there exist 10 on-line questionnaires designed by GVU. These questionnaires resemble different design characteristics such as linearity, adaptability, and generality. For example, the first survey is designed to be adaptive; that is, the order of the questions change with respect to the answers obtained from the user. As a second example, the tenth questionnaire is linear, non-adaptive, which is aimed to interpret the results as well as understanding the codes used in the data sets.

In order to illustrate the aspects of the GVU surveys, the following are the question categories that are used in the GVU's 10th WWW User Survey: general demographics, technology demographics, online privacy and security, Web and Internet use, software filters and content rating, everyday life, electronic commerce, and also a specific questionnaire for webmasters. As the survey consists of a broader range of question categories, it offers information about different aspects of the WWW usage.

For example for the survey on the Web preference, 3291 users are chosen and categorized as novice, intermediate, experienced and expert according to their responses to the pre-questionnaires.  Among these users, it is reported that users prefer using the web to watching TV, using a phone, sleeping, exercising, reading, watching movies, socializing or doing household work in their daily lives. Watching TV is least preferred (8.7%) as compared to exercising (32.5%).

The following question and the figure are included in the 10th WWW User Survey. They are copied here from the web site: http://www.gvu.gatech.edu/user_surveys/survey-1998-10/graphs/use/q01.htm.

Question: On average, how often do you use your WWW browser?

  • More than 9 times/day
  • 5 to 8 times/day
  • 1 to 4 times/day
  • A few times a week
  • Once a week
  • Once a month

The following figure illustrates one set of the results from the responses of the users to the previous question.

In this figure, it has been shown that the frequency of access to the WWW is different for different age groups. As it can be seen, users in the age groups 11-20 and 50+ are accessing WWW more frequently than those in the age group 21-50.

3 Survey Methods: Interviews

Interviews are among the most challenging and rewarding forms of measurement. They require a personal sensitivity and adaptability as well as the ability to stay within the bounds of the design of the interview.

Interviews are a popular HCI research method because they are flexible and participatory. Interviews are flexible because the interviewer has the freedom to change some questions or the asking order of the questions according to the reactions of the users. Finally, interviews are participatory since they require both the interviewer and the participant to join in an interactive conversation. This is a big advantage when compared with the isolated effect of the questionnaires because the user shares the experience and (s)he may have more tendencies to use the interface after the interview. Due to these reasons, interviews are more personal ways of gathering information from users than the questionnaires.

The most important types of interviews are face-to-face interviews and telephone interviews. In face-to-face interviews, the interviewer works directly with the respondent. Unlike questionnaires, the interviewer has the opportunity to monitor the user and ask follow-up questions. However, face-to-face interviews are rather time consuming due to some factors such as warming up time for the respondent, irrelevant conversations, etc. On the other hand, telephone interviews enable a researcher to gather information rapidly. Like face-to-face interviews, they allow for some personal contact between the interviewer and the user. But they also have some major disadvantages. For example, people often don't like the intrusion of a call to their homes. And, telephone interviews have to be relatively short or people may feel imposed upon. For a more detailed discussion on types of interviews, the reader is strongly recommended to read [Shaughnessy, et al., 1997].

Mainly, there are three methods that are used in designing the interviews in HCI research [Nielsen, 1993][Preece, et al., 1994]. Unstructured interviewing methods are used during the earlier stages of usability evaluation. The objective of the interviewer at this stage is to gather as much information as possible concerning the user's experience and on their expectations of the system. Semi-structured interviews are used when the interviewer has a better understanding of system requirements. Therefore, a more focused interview design can be used to focus on the points of interest. However, there can still be a degree of flexibility to allow the user to expand on an answer. Finally, structured interviewing has a specific, predetermined agenda with specific questions to guide and direct the interview. The interviewer, in this design, has a fully developed product and prepares questions to measure the user's reactions to that product.

Regardless of the interview method chosen, there are some issues that must be considered during the design of an interview from the very beginning to the very end [Trochim, 2001]. The first issue to consider is training of the interviewer, which is very important because the interviewer controls everything in an interview. Another important point is that s(he) should clarify any confusion that the users may have, and should answer the users’ questions clearly and honestly. The interviewer has the advantage of observing the users and acting accordingly. S(he) should control the mood of the interview by his/her behaviors, voice tone, etc.

Interviews can be considered to be similar to ethnographic methods. Both are data-gathering techniques involving user participation and both may be flexible. On the other hand, there are significant differences as well. Ethnographic methods, originated from anthropology, are qualitative data gathering methods, which require long-term observation of a group of people in their natural environments. Ethnographic methods can be used to help product design in HCI research. However, traditional ethnographic methods usually have long time requirements to be used in this field. Instead, relaxed forms of ethnographic methods such as Quick-and-dirty Ethnography, Contextual Inquiry, Video re-presentation, and Rapid Ethnography [Newman, 1999] [Millen, 2000] are used in HCI research. On the other hand, interviews are relatively short, and the researcher can collect quantitative data as well as qualitative data. Also, depending on the type of the interview, the researcher may not be able to establish the natural environments of the participants.

3.1 Interviews CASE STUDY: Exploring the Role of Metaphor in HCI

This work is concerned with the use of metaphors in the human-computer interface and their role in the relationship between the user and the computer. Interested reader is strongly recommended to consult to the actual manuscript presented at http://www.redwines.btinternet.co.uk/chris/phd.html.

In this work, interviews are used to explore the role of metaphor in the user's motivation and work effectiveness in user interface design. It is reported that unstructured and semi-structured interviews are used. This is because the interviews should be user-directed in order to determine the significance of the interface and the metaphors in it to the user. While testing the interface, the interviewer and the users are engaged in a conversation. The users are asked the following types of questions:

  • Why?

  • What for?    

  • Why is that?

  • Why do you use that?

  •  What do you do that for?

It is reported that there is a difference between these two questions in some circumstances. 'What?' implies an object, action or concept; the objects on the interfaces in this case. On the other hand, ‘Why?' implies a mechanism, in this case the signification.

It is reported in the paper that two interviews were carried out with designers of a manufacturing system in the pilot study. The users’ answers to the above questions are subjectively interpreted and scaled into 18 levels (i.e, 1-18) that are called significance levels. These levels are defined as follows: when a respondent has answered that A is done because of B, then B represents a higher level of signification.  These levels are defined by the interviewer before going into the interview. The interpretation of the answers was done and categorized in to the significance levels by using content analysis techniques, which are described in the paper. After all the interviews are completed, graphs showing that the users are categorized in higher significance levels with the interface having metaphors.

Here, we summarized the interviewing method that was used in this work. For a detailed discussion, please consult to http://www.redwines.btinternet.co.uk/chris/phd.html.

4 Conclusion and Recommendations

Survey research is important in HCI research because it helps in gathering information from the users about the design and development of the interface/product. In this text, we described two forms of surveys that are very commonly used for these purposes; namely questionnaires and interviews. Questionnaires usually provide quantitative data whereas interviews provide qualitative data. For this reason, questionnaires are more appropriate for making statistical analysis. On the other hand, interviews establish a rather warm atmosphere for the users so the users become more enthusiastic about using/testing the interface/product.

Since a questionnaire provides quantitative data, there may be situations in which it must be supported by interviews to get qualitative data about the users such as their mood, their willingness to participate to the research, external factors such as the environment in which users live, etc. Understanding user's psychology can be helpful for interpreting the statistical results correctly. For this purpose, even ethnographic methods can be used provided that the researcher can set up the necessary environment.

All of the techniques described in this text are quite general; the reader has to re-interpret and tailor each survey-research method to suit the kind of activity that he/she is doing.

References

[Brehob, 2001] Brehob, K., et al. “Usability Glossary,” http://www.usabilityfirst.com, 2001.

[Hom, 1998] Hom, J., “The Usability Methods ToolBox,” http://www.best.com/~jthom/usability/, 1998.

[Kirakowski, 1998] Kirakowski, J., “Questionnaires in Usability engineering,” http://www.ucc.ie/hfrg/resources/qfaq1.html, 1998.

[Millen, 2000] Millen, D.R., "Rapid Ethnography: Time Deepening Strategies for HCI Field Research," pp. 280-286, ACM Press , NY , USA , 2000. 

[Newman, 1999] Newman, M., "An Investigation into Web Site Design Practice," U.C. Berkeley MIG Seminar, 1999.

[Nielsen, 1993] Nielsen, J., "Usability Engineering" pp.209-214, Academic Press, 1993.

[Perlman, 1998] Pearlman, G., “Web-Based User Interface Evaluation with Questionnaires,” http://www.cs.umd.edu/~zzj/FramedLi.htm?http://www.acm.org/~perlman/question.html, 1998.

[Preece, et al., 1994] Preece, J., and et al. "Human-Computer Interaction," Addison-Wesley Longman Limited, 1994.

[Shaughnessy, et al., 1997] Shaughnessy, J.J., Zechmeister, E.B., "Research Methods in Psychology," Fourth Edition, McGraw Hill, 1997.

[Shneiderman, 1998] Shneiderman, B.,"Designing the User Interface," Third Edition, Addison Wesley Longman, Inc. 1998.

[Trochim, 2000] Trochim, W.M., “Research Methods Knowledge Base – Survey Research,” http://trochim.human.cornell.edu/kb/survey.htm, 2000.

* For any comments please contact to Cemal Yilmaz or Ugur Kuter