Introduction

The Role of Theory
in HCI


Development Methods

Ethnographic
Methods


Controlled
Experimentation


Survey Methods

Logging &
Automated Metrics



CHARM
Choosing Human-Computer Interaction (HCI) Appropriate Research Methods




Ethnographic Methods

Jason Burke (burkej@isr.umd.edu)
Institute for Systems Research

Andrea Kirk (kirkam@wam.umd.edu)
Department of Computer Science

University of Maryland 
College Park, MD 20742 USA
October 2001

 

1. Introduction

Ethnography is in itself not so much a method as a category of human-computer interaction research.  This kind of research has been adapted from sociology and anthropology, where it is a method of observing human interactions in social settings and activities.  It can also be described as the observation of people in their cultural context.  A culture is defined by Massey (1998) as being "...made up of certain values, practices, relationships and identifications."  Thus, one can describe a workplace as a culture, filled with work standards, business practices (both formal and informal), and relationships between coworkers and between workers and managers.

Why should user interface designers be concerned with the user culture?  There are several reasons why ethnography is of vital importance to good interface design, including these:

  • An ethnographic study is a powerful assessment of users' needs: A crucial goal of an ethnographic study is to gain the capacity to view a system through the eyes of the user.  This perspective is extremely useful in creating a user interface to fit the needs of the end-user.
      
  • It uncovers the true nature of the system user's job: A goal of an ethnographic study is to uncover all tasks and relationships that combine to form a user's job. It is often the case that a user performs tasks and communicates in ways that are outside of their official job description.
      
  • The ethnographer can play the role of the end-user:  The high level of user understanding that an ethnographer can gain through his/her fieldwork can be a useful bonus.  For example, the ethnographer can act as the end-user in participatory design when "real" end-users are difficult to procure.
      
  • The open-ended and unbiased nature of ethnography allows for discovery:  Other HCI research methods, such as task analysis and controlled experimentation, must formalize, categorize, and/or theorize how users interact with a system in order to yield quantitative results.  The unassuming nature of ethnography can often yield unexpected revelations about how a system is used.

As would be expected, however, there are also some drawbacks to using ethnography, including:

  • Time requirements: The rewards that come from an ethnographic study are directly related to the time investment.  While external constraints often limit field studies to a few days or hours, formal ethnographic studies have been known to take weeks or even months.
      
  • Presentation of results: The highly qualitative nature of results can make them difficult to present in a manner that is usable by designers.
      
  • Scale: Most ethnographic studies use a small number of participants and a small-scale environment (Hughes et al., 1995).  Increasing the scale can be extremely difficult as it imposes a much greater amount of cost, communication, and effort.

Visual observation is not the only way to gather data in ethnographic studies.  For an in-depth understanding of the user culture, the researcher should watch, participate in and inquire about the users' normal activities.  Like in sociological studies, the researchers should be immersed in the culture to better understand what is going on around them.  From a design perspective, it's necessary to think like a user (or at least understand how the users think) to create a better interface.

There are many terms used in HCI/usability that are related, if not synonymous, to ethnography.  The term field study is generally used interchangeably with ethnography.  Contextual inquiry is a slightly more specific form of ethnography focusing on asking questions hence "inquiry" (Rose et al., 1995).  Observational study entails simply watching users, without asking questions of why or how things are being done.  Participant observation focuses on the ethnographer joining in user activities to better understand the processes involved.  The latter three activities used together form the basis of a thorough ethnographic study.

The goal of sociological ethnography is to understand an individual's or group's interactions within the culture.  The data gained from ethnographical research is almost entirely qualitative.  The goal of ethnography for systems designers, on the other hand, is the improvement of a system in use in the culture by finding the problems in the way it is currently used.  This involves observing users' interactions with the system as well as the effects the system has on the culture of the workplace.  McCleverty (1997) states, "The goal of an ethnographic study is to identify routine practices, problems, and possibilities for development within a given activity or setting." 

The majority of the data in an ethnographic study is elicited through the process of in-person observation, audio/video observation, and interviews.  Thus, the results of an ethnographic study are largely qualitative (impressions, opinions, environment descriptions, etc.).  Error rates and questionnaire results are examples of the types of quantitative data that can be compiled in an ethnographic study.

The following is a sample of the data obtained from a study done by Anne McClard and Patricia Somers (2000) on an Internet-capable tablet device.  Both qualitative data, gathered by formal and informal interviews, and quantitative data from logging the use of the devices, were obtained in this experiment.

Qualitative (inquiry)

"Three households reported using chat between the PC and the tablet.  One example of this is interesting because it illustrates a creative simultaneous use of the PC and the tablet to accomplish a task.  From the tracking data, we noticed that one household was simultaneously looking at the same web site on both the tablet and the PC while using chat.  When we asked what was going on, it turned out that two household members were looking at www.realtor.com together to find a house to buy.  Even though their PC was located in the dining area adjoining their living room and they were only feet away from each other, they were using chat to send each other URLs to look at.  At the same time, they were vocally discussing the items they were viewing."

Quantitative (traffic logging)

"On both the tablet and the PC, there was a great diversity of web sites visited.  On the tablet, 270 sites accounted for 75 percent of the time spent browsing, and 85 percent of the top 100 sites were visited by only one person.  Certain categories of web sites were popular across households (for example, six of the thirteen households visited financial sites regularly), but each household had its own favorite site within those categories."

Valuable data can be discovered through ethnographic methods that might never be found through interviews outside of the workplace, and certainly not through introspection or walkthrough methods done in-house by the interface designers.  For instance, while users may report that they use a system for all of their work, giving the impression that they find the system easy to work with, they may have instituted workarounds or ways to make do with a system that is not filling all of their needs (McCleverty, 1997).  Only by directly observing the users at work can designers capture these unspoken user needs.

Another useful result of an ethnographic study is that the individual(s) who conduct the study can provide a user's expertise to the rest of the team during implementation and evaluation.  After having observed real users in their real environment, and perhaps having joined the users in their tasks, the researchers obtain a firsthand sense of the needs and concerns of the users.  They can help the design team find weak points in an interface and suggest ideal improvements that would benefit the users most.

Even with these specific goals in mind, the results of an ethnographic study are hard to quantify, as mentioned above.  Put in another way, sociologists want to analyze, while designers want to synthesize (McCleverty, 1997).  The sociologist may simply want data for knowledge's sake; the designer wants to use the data to make decisions to improve an existing system or to create a new one.  Nonetheless, it is becoming more clear in today's world of user interface design that the user's environment must be understood in order to create a better interface.

The following texts are cited as excellent sources of information regarding traditional ethnography.  A firm grasp of the fundamentals in ethnographic theory can help the researcher become adept in applying ethnographic methods to support user interface design.

"Ethnography: Step by Step" by David M. Fetterman

--  

In his book, Fetterman covers the nature of fieldwork, the equipment needed to conduct research, the analysis of data, the differences and similarities between qualitative and quantitative approaches, and writing the report.
  
"Ethnography: A Way of Seeing" by Harry F. Wolcott

--  

This book goes beyond defining ethnography as a set of field methods and practices.  Walcott distinguishes ethnography as "a way of seeing" through the lens of culture and provides guidance to readers for making their study more ethnographic. 

 

2. Methods

There are various methods by which ethnography can be incorporated into user interface design.  The main approaches in use today are listed below.

2.1. Concurrent Ethnography

This is the most famous method of ethnography used in HCI research. It is also sometimes referred to as informed ethnography or ethnomethodology.  This method implies that an ethnographic study is being performed in support of and at the same time as the design of a new system. Usually, a system prototype is developed and refined based upon the results of the study.

Advantages: 
It ensures focus upon the user through all stages of new system development.

Disadvantages: 
In comparison to the other ethnographic methods, concurrent ethnography requires the largest amount of cooperation, coordination, and time.

Examples/Links:
"Moving out from the control room: Ethnography in system design."

--  

Research paper:  provides a description and an example of concurrent ethnography in air traffic control.
http://delivery.acm.org/10.1145/200000/193065/p429-hughes.pdf
  
"Ethnographically-informed systems design for air traffic control."

--  

Research paper:  provides an overview of an informed ethnography study in air traffic control.
http://delivery.acm.org/10.1145/150000/143470/p123-bentley.pdf


2.2. Evaluative Ethnography
  

The goal of this method is to evaluate a new design model.  Ethnographic research is performed within a narrow context that is aimed at targeting the aspects of work that would be affected by the new design model.

Advantages: 
It is useful in helping to prove or disprove a new design model or theory.  The restricted domain allows an evaluative ethnographic study to be performed in a relatively short time frame.

Disadvantages:
The tight focus can blind the ethnographer to important information that is outside the domain of the study.

Examples/Links:
"Moving out from the control room: Ethnography in system design."

--  

Research paper:  provides a description and an example of concurrent ethnography in air traffic control.
http://delivery.acm.org/10.1145/200000/193065/p429-hughes.pdf


2.3. Quick and Dirty Method

This method is often used as a precursor to other ethnographic research methods.  It can be useful in increasing awareness of large-scale usability and acceptability issues that are important in the design of a new system.  However, it is frequently the only form of ethnography practiced due to imposing time and/or budget constraints. 

Advantages: 
It can yield valuable knowledge of the social organization within a work setting in a short amount of time relative to the size of the project (Hughes et al., 1995). 

Disadvantages: 
The results are limited to a general understanding of a work culture.  It is possible that a quick and dirty ethnographic study can lead to a false sense of understanding for a working culture.

Examples/Links:
"Working with 'Constant Interruption': CSCW and the Small Office."

--  

Research paper:  A study done using the "quick and dirty" method.
http://delivery.acm.org/10.1145/200000/193028/p275-rouncefield.pdf


2.4. Rapid Ethnography

This is the newest of the ethnographic methods in practice.  It uses "a collection of field methods intended to provide a reasonable understanding of users and their activities given significant time pressures and limited time in the field" (Millen, 2000).  Some of the key tenets of this practice are to use a constrained focus, key informants, and multiple ethnographic observers.

Advantages: 
This method uses a short time frame on par with the quick and dirty method.  However, it is more rigorous and formalized than the quick and dirty method.

Disadvantages: 
The process is still undergoing refinement.  Choosing "key" informants and a constrained focus can prove to be difficult.

Examples/Links:
"Rapid Ethnography:  Time Deepening Strategies for HCI Field Research."

--  

Research paper:  Contains a description of the rapid ethnography procedure and a short case study.
http://delivery.acm.org/10.1145/350000/347763/p280-millen.pdf


2.5. Reexamining previous studies

Many ethnographic studies have already been done in many areas.  While it is unlikely to find a one that is completely suitable, previous studies that are even remotely relevant to the application at hand can be a "useful way of sensitizing designers to the social organization character of a considerable variety of settings" (Hughes et al., 1995).

Advantages:
Compared to the other ethnographic methods,
reexamining previous studies requires the least amount of time and money.

Disadvantages: 
It is unlikely to find a previous study that is completely suitable to the desired domain.

Examples/Links:
"The Role of Ethnography in Interactive Systems Design."

--  

Research paper:  Contains a description of reexamining previous studies in ethnography and a short example.
http://delivery.acm.org/10.1145/210000/205358/p56-hughes.pdf 

 

3. Guidelines and Recommendations

In order to facilitate the generation of useful data, the ethnographer should understand the basic steps that are used to conduct an ethnographic study.  The following guidelines are recommended as a general framework for an ethnographic study (Rose et al., 1995):

Preparation

  • Understand organization policies and work culture.

  • Familiarize yourself with the system and its history.

  • Set initial goals and prepare questions.

  • Gain access and permission to observe/interview.

Field Study

  • Establish rapport with managers and users.

  • Observe/interview users in their workplace and collect subjective/objective quantitative/qualitative data.

  • Follow any leads that emerge from the visits.

  • Record your visits.

Analysis

  • Compile the collected data in numerical, textual, and multimedia databases.

  • Quantify data and compile statistics.

  • Reduce and interpret the data.

  • Refine the goals and the process used.

Reporting

  • Consider multiple audiences and goals.

  • Prepare a report and present the findings.

In conclusion, the authors of this paper would like to make the following suggestions regarding the use of ethnographic research to support user interface design:

  • Use ethnographic methods early in the design process:  The high risk of designing a system for an unknown or misunderstood user warrants the time investment in ethnographic methods.
      

  • Have a well-defined scope: The open-ended nature of ethnography can be a weakness as well as a strength.  It is important to stay focused and to recognize what users are necessary to include in an ethnographic study.
      

  • Choose a proper level of ethnographic study: Time frame and budget constraints will often dictate what level and type of ethnographic methods should be used.
      

  • Make use of previous ethnographic studies: This should be a precursor to any field study as it requires the least amount of resource expenditure.
      

  • Wear one hat at a time: Trying to play the role of the role of the ethnographer and the designer at the same time can be hazardous.  Make sure to understand the user within the context of the current system before attempting to make design changes.

 

4. References

Bentley, R., J. A. Hughes, D. Randall, T. Rodden, P. Sawyer, D. Shapiro, and I. Sommerville. "Ethnographically-Informed Systems Design for Air Traffic Control."  Proceedings of the ACM 1992 conference for Computer Supported Cooperative Work, ed. J. Turner and R. Kraut. New York, NY: ACM Press, 1992, pp. 123-129.  http://delivery.acm.org/10.1145/150000/143470/p123-bentley.pdf

Fetterman, David M. "Ethnography: Step by Step." Applied Social Research Methods Series, Vol. 17, Walnut Creek, CA: Sage Publications, Inc., 1998.

Hom, James.  "The Usability Methods Toolbox." Website.  http://www.best.com/~jthom/usability/.  1996.

Hughes, J. A., King, V., Rodden, T., and Andersen, H.  "Moving out from the control room: Ethnography in system design". Proceedings of the ACM 1994 Conference for Computer Supported Cooperative Work. New York, NY: ACM Press, 1994, pp. 429-439. http://delivery.acm.org/10.1145/200000/193065/p429-hughes.pdf 

Hughes, J., King, V., Rodden, T., and Anderson, H. "The role of ethnography in interactive systems design." Interactions, 2, 2, 1995: pp. 56-65.  http://delivery.acm.org/10.1145/210000/205358/p56-hughes.pdf 

Massey, Alexander.  "'The Way We Do Things Around Here': The Culture of Ethnography." Website. http://www.geocities.com/Tokyo/2961/waywedo.htm. 1998.  

McClard, Anne, and Patricia Somers.  "Unleashed: Web tablet integration into the home."  Proceedings of the CHI 2000 conference on human factors in computing systems.  New York, NY: ACM Press, 2000, pp. 1-8.  http://delivery.acm.org/10.1145/340000/332042/p1-mcclard.pdf 

McCleverty, Amy.  "Ethnography." Website.
http://www.cpsc.ucalgary.ca/~saul/681/1997/amy/ethnography.html. 1997.

Millen, David.  "Rapid Ethnography:  Time Deepening Strategies for HCI Field Research."  Proceedings of the ACM 2000 conference for Designing interactive systems: processes, practices, methods, and techniques.  New York, NY: ACM Press, 2000, pp. 280-286.  http://delivery.acm.org/10.1145/350000/347763/p280-millen.pdf   

Perlman, Gary.  "HCI Bibliography: Free Access to Human-Computer Interaction Resources."  Website.  http://www.hcibib.org/. 2001.

Rose, Anne, Plaisant, Catherine, and Shneiderman, Ben. "Using Ethnographic Methods In User Interface Re-engineering." Proceedings of the ACM 1995 conference for Designing Interactive Systems: processes, practices, methods, and techniques.  New York, NY: ACM Press, 1995, pp. 115-122.  http://delivery.acm.org/10.1145/230000/225447/p115-rose.pdf

Rouncefield, M., J.A. Hughes, T. Rodden, S. Viller.  "Working with 'constant interruption': CSCW and the Small Office." Proceedings of the ACM 1994 Conference on Computer Supported Cooperative Work.  New York, NY: ACM Press, 1994, pp. 275-286.  http://delivery.acm.org/10.1145/200000/193028/p275-rouncefield.pdf  

Wolcott, Harry F.  "Ethnography: A Way of Seeing." Walnut Creek, CA:  AltaMira Press, 1999.

 

5. Acknowledgements

Special thanks to Diane Maloney-Krichmar for providing helpful comments and contributing the Fetterman and Wolcott reference descriptions.



Page Last Updated:  November 20, 2001