Key Concepts
Survey research is one of the most common forms of research.
Survey research generally involves asking people about a topic, although some surveys may be based on existing data sources.
A clear research question and specific objectives help guide the design of a survey and questionnaire.
Survey topics may need to be investigated with focus groups in order to know the specific issues that are important to study.
Survey methods are determined by the research question. They include interviews and self-administered questionnaires using the mail or telephone. They may be web-based, or given in person.
The most appropriate types of questions and response scales should be selected, and questions should be free from bias.
Questions that have a predetermined list of possible responses are easier to analyze than open-ended questions.
The choice of scale (nominal, ordinal, numerical) has ramifications for ways in which the results can be analyzed.
Answers need to be balanced with equal numbers of positive and negative response options.
In general, responses should be specific and avoid vague adjectives.
Response categories need to be homogeneous and mutually exclusive.
Potentially objectionable questions can sometimes be softened so they are more acceptable to the subject.
Questions permitting subjects to select as many options as they choose can present challenges at the time of analysis.
People often find ranking scales difficult if more than three choices are to be ranked.
Likert scales are popular and easy for respondents to understand.
Reliability indicates whether the results from a test or questionnaire are reproducible.
Validity indicates whether a test or questionnaire is measuring what it is intended to measure.
Pilot testing is essential and need not use a large sample.
A high response rate provides more confidence in the findings.
Effective follow-up is the most important way to increase response rates.
A well-crafted cover letter should accompany all questionnaires.
Incentives, both monetary and nonmonetary, appear to have a large effect on response rates.
Ethical considerations and institutional review boards require that respondents be assured that their responses will be kept confidential.
Appropriate sampling methods lead to more valid studies so long as the type of respondent is pertinent to the research question.
Determining the required number of subjects is part of the survey design process.
Survey results are analyzed using the methods previously discussed in this book.
Presenting Problems
Fellowship and residency program directors may want to know which attributes of their program attract applicants. It can be difficult for applicants to choose among programs, and certain factors are more important to some applicants than others. Program directors want to highlight the features of their program that are attractive to the kinds of applicants they would like to have. Caiola and Litaker (2000) wanted to learn more about the factors that appeal to internal medicine residents when choosing a fellowship in general internal medicine (GIM). A group of faculty at their institution developed a 36-item questionnaire to learn how the interview process, program location, and family considerations affected fellowship selection. We discuss how Caiola and Litaker selected the sample of people to complete their questionnaire, how they designed the questionnaire, and their findings. Data are on the CD-ROM [available only with the book] in a folder called “Caiola.”
Numerous studies have demonstrated the protective effect of exercise against many chronic diseases including heart disease, hypertension, diabetes mellitus, stroke, and osteoporosis. Physicians have an important role in educating their patients about the benefits of regular exercise.
Rogers and colleagues (2002) wanted to determine the percentage of internal medicine residents who counseled a majority of their patients about exercise and factors that influenced their exercise counseling behavior. They sent a self-administered questionnaire to all internal medicine residents at six U.S. training programs. Data collected included demographic information, counseling practices, perceived exercise benefits, attitude toward counseling, barriers to counseling, and personal exercise habits. We discuss the response rate they obtained in their study, how they selected the sample, and their conclusions, using data on the CD-ROM [available only with the book] in a folder called “Rogers.”
A study by Lapidus and his colleagues (2002) was a presenting problem in Chapters 3 and 6. Recall that they undertook a survey to assess domestic violence (DV) education and training and whether pediatricians and family physicians screen emergency department patients for DV. Domestic violence was defined as “past or current physical, sexual, emotional, or verbal harm to a woman caused by a spouse, partner, or family member.” Please see Chapters 3 and 6 for more detail. We describe how they obtained their sample and some of their findings using data on the CD-ROM [available only with the book] in the folder called “Lapidus.”
Urinary tract infections (UTIs) are among the most common bacterial infections, accounting for 7 million outpatient visits and 1 million hospital admissions in the United States each year. They are usually caused by Escherichia coli and are often treated empirically with trimethoprim–sulfamethoxazole. Recent guidelines from the Infectious Disease Society of America recommend this drug as standard therapy with specific fluoroquinolones as second-line choices. Two important factors in the selection of treatment are the health care costs of drug therapy and emergence of resistance to trimethoprim–sulfamethoxazole among E. coli strains causing UTIs. Huang and Stafford (2002) used survey data from the National Ambulatory Medical Care Survey (NAMCS) to assess the demographics and clinical characteristics of women who visit primary care physicians and specialists for UTIs.
The Research Questions
With a little common sense and the information from this chapter, you can develop questionnaires to evaluate courses, learn student preferences for use in an educational program, and other undertakings in which you want to gather information locally. For more extensive projects, especially those related to a research project, you may wish to consult a biostatistician, epidemiologist, or other professional with survey expertise.
Everyone knows about surveys. In fact, the average person’s opinion of statistics is greatly based on the results of polls and marketing surveys. Many people associate surveys with the U.S. census, which occurs every 10 years, but a census is not really a survey. A census queries everyone in the population, whereas a survey queries only a sample from the population. Survey research became more prominent after World War II as manufacturers began to market products to the public and hired marketing experts to learn what would make the product sell. At the same time, political pollsters began to have an increasing presence garnering public opinion about specific issues and learning which candidate was leading during an election campaign. Today surveys seem omnipresent, and, as a result, many people think that survey research is easy—just design some questions and get some people to answer them. The purpose of this chapter is to acquaint you with some of the issues in designing, administering, and interpreting surveys.
The term poll is generally associated with political polls, but it simply means asking questions in order to learn information or opinions. The Gallup Organization and the Roper Center are two of the largest polling organizations in the United States. They both use proper methods to design surveys and to select random samples of respondents.
A large number of excellent resources exist for those who want to design a survey. We have selected some salient issues to discuss in this chapter, and we have used a number of resources which you may wish to consult if you desire further information: Dillman (2000), Fink and Kosecoff (1998), Litwin and Fink (1995), Korn and Graubard (1999), and Rea and Parker (1997).
The need to know the answer to a question generally motivates survey research, such as those in the Presenting Problems. As with any other research question, a crucial first step is to review the literature to learn what is known on the topic and what other methods have been used previously.
It may be difficult to specify the precise issues that should be addressed in a survey. For instance, suppose a researcher wants to know how health care workers view the elderly, and after performing a MEDLINE search, can find little information on the subject. In this situation, focus groups may help refine the issues. Focus groups are interviews of a small number of people, generally 6–10, rather than individual interviews, and are done in person or over the telephone. One can get a great deal of information during a focus group session by obtaining in-depth responses to a few general questions. For information about focus groups, see Kruger and Casey (2000) and Morgan (1997). Of course, asking an expert, if one is available, is always advisable—in developing a research question as well as in reviewing a draft questionnaire.
Most surveys use self-administered questionnaires—either in person or via mail or email, or interviews—again in person or over the telephone. Advantages and disadvantages exist for each method, some of which are illustrated in Table 11–1.
| Self-Administered Mail/Email | Self-Administered in Person | Interview by Phone | Interview in Person | |
|---|---|---|---|---|
| Cost | ++ | + | – | – |
| Time | ++ | + | – | – |
| Standardization | + | + | +/– | +/– |
| Depth/detail | – | – | + | ++ |
| Response rate | – | ++ | + | ++ |
| Missing responses | – | ++ | ++ | ++ |
When interviews are used for survey research, they are often called structured interviews, in which the same questions are administered to each subject in the same order, and no coaching is permitted. In other words, the researcher tries to make the questions and the manner in which they are asked identical for all persons being interviewed. In other situations, less-structured interviews that permit the investigator to probe areas can yield rich results, although the data do not lend themselves to the quantitative methods we have used in this book. A famous example was the pioneering work in human sexuality by Alfred Kinsey in the 1940s in which he interviewed many people to learn about their sexual habits. Book 4 in the Survey Kit by Litwin and Fink (1995) is devoted to how to conduct interviews in person and by telephone.
The majority of the studies in the medical literature use self-administered questionnaires, as did the investigators in the first three Presenting Problems. Book 3 in the Survey Kit by Litwin and Fink (1995) discusses how to conduct self-administered and mail surveys. Dillman (2000) covers email and Internet surveys. Researchers are also turning to national data bases to answer survey questions, such as the study by Huang and Stafford (2002).
Surveys are generally thought of as cross-sectional and measuring the current situation, and certainly most surveys fit this category. Questionnaires are used in cohort and case–control studies as well, however. The Nurses’ Health study is a good example; this study began in the late 1970s with follow-up questionnaires mailed to over 115,000 nurses every 2 years (Colditz et al, 1997).
Many ways to ask questions exist, and survey designers want to state questions so that they are clear and obtain the desired information. We discuss a number of examples, some of which are modeled on the text by Dillman (2000) and subsequently modified to reflect medical content by Dr. Laura Q. Rogers.
Open-ended questions are ones that permit the subject to respond in his or her own words. They are sometimes appropriate, as shown in Table 11–2, such as when the topic has not been studied before, and it is important to obtain answers that have not been cued by any listed responses. Many researchers analyze the content of the answers and try to classify them into categories. A primary advantage of open-ended questions is the capability to report some of the prototypic answers using a subject’s own words. Closed-response questions are more difficult to write, but their payoff is the ease with which the answers can be analyzed and reported. Many of the statistical methods discussed in this text can be used to analyze survey responses. A compromise approach can be taken by using a set of responses gleaned from the literature or focus groups, but also adding an opportunity for subjects to write in a brief answer if none from the responses fits. These questions generally have an “Other—please explain” at the end of the response set.
| Use Open | Use Closed | |
|---|---|---|
| Purpose | Actual words or quotes | Most common answers |
| Respondents | Capable of providing answers in their words | Willing to answer only if easy and quick; may write illegibly |
| Asking the question | Choices are unknown | Choices can be anticipated |
| Analyzing results | Content analysis; time-consuming | Counting or scoring |
| Reporting results | Individual or grouped responses | Statistical data |
An example of a question with open versus closed questions is given in Box 11–1. Does the closed response set have an answer with which you would be comfortable?
| Open Question |
| What is the importance of reputation in choosing a fellowship? |
| Please describe: _______________________________ |
| Closed question |
| What is the importance of reputation in choosing a fellowship? |
| □ Definitely important |
| □ Probably important |
| □ Probably unimportant |
| □ Definitely unimportant |
If closed questions are used, the researcher needs to decide the level of detail required in the answers. The scale helps determine which methods can be used to analyze the results. Sometimes knowing only whether a subject agrees with the question is sufficient (nominal), such as whether a large city location is important in the choice of a training program (Box 11–2). An ordinal scale may be used if knowing gradations of importance is desired. Alternatively, an ordinal scale giving a choice of city sizes provides more specific information. At the other end of the spectrum is an open-ended question in which the subject provides a number (numerical). Depending on the question, we generally recommend that researchers collect data in as much detail as is reasonable; for instance, it is always possible to form categories of age for tables or graphs later (see Chapter 3). On the other hand, too much detail is not necessary and leads to unreliable data. For the question about city size, we would opt for one of the two ordinal response sets.
| Nominal |
| Is a large city environment important in choosing a fellowship? |
| □ Yes |
| □ No |
| Ordinal |
| How important is a large city environment in choosing a fellowship? |
| □ Definitely important |
| □ Somewhat important |
| □ Doesn’t matter |
| □ Somewhat unimportant |
| □ Definitely unimportant |
| How large a city is important in choosing a fellowship? |
| □ 25,000 or less |
| □ 25,001 to 50,000 |
| □ 50,001 to 100,000 |
| □ 100,001 to 500,000 |
| □ Larger than 500,000 |
| Numerical |
| How large a city is important in choosing a fellowship? |
| _____________________________________________ |
Questions using an ordinal response should provide as many positive as negative options. Box 11–3 shows two sets of options regarding students’ satisfaction with the Introduction to Clinical Medicine course. The first set places the neutral answer in the fourth position and provides only one answer in which students can express dissatisfaction. The second set is more likely to provide valid responses, with the neutral answer in the middle position and the same number of positive and negative options.
| Problem: |
| How satisfied were you with the Introduction to Clinical Medicine Course? |
| □ Completely satisfied |
| □ Mostly satisfied |
| □ Somewhat satisfied |
| □ Neither satisfied or dissatisfied |
| □ Dissatisfied |
| Revision to balance responses: |
| How satisfied were you with the Introduction to Clinical Medicine Course? |
| □ Completely satisfied |
| □ Mostly satisfied |
| □ Neither satisfied or dissatisfied |
| □ Mostly dissatisfied |
| □ Completely dissatisfied |
What are the potential problems in using terms like sometimes, often, rarely, etc? Box 11–4 illustrates two questions that ask the amount of alcohol consumed weekly. The choices specifying actual numbers of drinks is much less ambiguous and will be easier to interpret.
| Problem: |
| How many alcoholic beverages did you drink during the past week? |
| □ None |
| □ A few |
| □ Several |
| □ Many |
| Revision to be less vague: |
| How many alcoholic beverages did you drink during the past week? |
| □ Zero |
| □ 1–5 |
| □ 6–10 |
| □ 11–20 |
| □ > 20 |
In filling out a questionnaire, how many times have you encountered a question in which you don’t understand the issue being addressed? This may result because the options are not mutually exclusive. For example, the question in Box 11–5 is confusing because it is mixing educational activities and sources of information. The easiest solution is to form two questions, each of which has mutually exclusive answers.
| Problem: |
| Which of the following sources provided you with the most information concerning smoking cessation? |
| □ Medical school training |
| □ Residency training |
| □ Attending physician |
| □ Colleague |
| □ Printed medical journal |
| □ Web site or other web-related resource |
| □ Published book/pamphlet |
| □ Drug salesperson |
| Revision to be mutually exclusive: |
| Which educational experience provided you with the most information concerning smoking cessation? |
| □ Medical school training |
| □ Residency training |
| □ Continuing education |
| Which source provided you with the most information concerning smoking cessation? |
| □ Colleague |
| □ Printed medical journal |
| □ Web site or other web-related resource |
| □ Published book/pamphlet |
| □ Drug salesperson |
Some questions are viewed as very personal, and people hesitate to divulge the information. Examples include income, sexual activity, and personal habits; see Box 11–6. These questions can be dealt with in two ways. First is to soften the manner in which the question is asked, perhaps by asking for less detail; it is generally better to obtain an approximate answer than none at all. Second is placement in the questionnaire itself; some survey experts recommend placement near the end of the questionnaire, after people have become comfortable with answering questions.
| Problem: |
| What was your total income from all sources in 2003? |
| $_______ Total income for 2003 |
| Softening objectional questions: |
| Which of the following broad categories best describes your total income from all sources in 2003? |
| □ $10,000 or less |
| □ $10,001 to $20,000 |
| □ $20,001 to $35,000 |
| □ $35,001 to $50,000 |
| □ $50,001 to $100,000 |
| □ More than $100,000 |
Items in which subjects can choose as many options as they wish are often used when survey questions ask about qualities of a product or why consumers selected a specific model, school, or service. They do not force the subject to single out one best feature. They can, however, be tricky to analyze. The best approach is to treat each option as a yes/no variable and calculate the percentage of time each was selected. Rogers and colleagues (2002) took this approach when they wanted to know which questions physicians routinely ask of patients who are not exercising adequately. Box 11–7 shows that how they used the responses of ask/don’t ask for each question.
For your patients who you believe do NOT exercise adequately, about which of the following aspects do you ask routinely? | ||
| Ask | Don’t Ask | |
| 10. Why they do not exercise | 1 | 0 |
| 11. What previous types of exercise they have done | 1 | 0 |
| 12. What their attitude is towards exercise | 1 | 0 |
| 13. What their beliefs are about exercise | 1 | 0 |
| 14. What type of exercise their friends or family get | 1 | 0 |
| 15. What they know about how to exercise to improve health | 1 | 0 |
No consensus exists in the use of a “Don’t Know” or “Undecided” category. Some researchers do not like to provide an opportunity for a subject not to commit to an answer. Others point out that not providing this type of option forces respondents to indicate opinions they do not really hold. If these categories are used, placement at the end of the question rather than in the middle of the question has been shown to increase the completion rate of questions by 9%.
