DATA COLLECTION

 

• Survey Research and Sampling

 

 

Introduction

 

This topic addresses survey research. It is not only widely used it als0 illustrates very well the processes and problems that result when research is taken out of the lab and into the real world. 

 

Survey Research and Sampling

 

Survey research (or “public opinion polling”) is another “scientific” approach to the study of human behavior.  However, it also serves to demonstrate ways in which social research is “as much an art as a science.”  Good survey research goes to great lengths to employ rigorous sampling methods.  Equal care is devoted to careful design and pre-testing of questionnaires, training of interviewers, and processing and analyzing data.  On the other hand, survey research involves a great deal of uncertainty, and requires making a number of judgment calls about which reasonable people will disagree.

 

Sampling is very important in social science research.  It is often impossible to study all of the cases we might wish to, and so we instead take a sample from a larger population (also called a universe).  Survey research is the most common, though not the only, social science application of sampling techniques.

 

Ideally, a sample should be random.  A random sample is one such that each item in the population from which the sample is drawn has an equal probability of being included in the sample[3] .  The reason why this is important is that a random sample provides an unbiased estimate of the characteristics of the population.  The sample can then be used to “represent” the population.  Put another way, if 60 percent of a random sample of voters favor candidate X then, although it would be impractical to interview all voters, our best guess is that 60 percent of all voters also favor candidate X. 

 

The reliability of this best guess will increase with the size of the sample.    If we have only interviewed a sample of 10 voters our results will be a lot less reliable than if we have interviewed a thousand.  Ninety-five times out of a hundred, a random sample of 1,000 will be accurate to within about 3 percentage points.  Put a bit more formally, such a sample has a “margin of error,” or “confidence interval” of approximately plus or minus 3 percent at a 95 percent “confidence level.”  If a random sample of 1,000 voters shows that 60 percent favor candidate X, there is a 95 percent chance that the real figure in the population is someplace in the range of about 57 to 63 percent.

 

Creative Research Systems has provided an elegant on-line sample size calculator. http://www.surveysystem.com/sscalc.htm.  In the first dialog box, select the 95% confidence level, enter 3 for the confidence interval and 20000 for the population, then click on “Calculate.”  What sample size do you need? Without changing anything else, add a zero to the population size, changing it 200000.  How much does the needed sample size increase?   Add three more zeroes, making the population size 200000000 (two hundred million).  By now you should have reached the conclusion that, beyond a certain point, the size of the population makes little difference.  If you were sampling students on your campus, you would need almost as large a sample as you would if your population consisted of all adults in the country.

 

In the second dialog box, select the 95% confidence level, enter 1000 as the sample size and 20000000 as the population.  (Leave “Percentage” at 50. This refers to the estimated percent of the population having the characteristic you are sampling, and 50 is the most conservative option.)  Click on “Calculate.”  What is the confidence interval?  Without changing anything else, double the sample size to 2000 and again click on “Calculate.”   Notice that the confidence interval is not reduced dramatically.  That is why most surveys don’t exceed more than about 1,500 respondents.  The number of interviews is the dominant factor in driving the costs of a survey, and beyond a certain point increasing this number is not cost effective, since costs will increase almost proportionately, but the margin of error will be reduced only a little.

 

Often it is not practical to carry out a pure random sample.  One common shortcut is the “area cluster sample.”  In this approach, a number of “Primary Sampling Units” (PSUs) are selected within a larger geographic area.  For example, a study of the United States might begin by choosing a subset of congressional districts.  Within each PSU, smaller areas may be selected in several stages down to the individual household.  Within each household, an individual respondent is then chosen.  Ideally, each stage of the process is carried out at random.  Even when this is done, the resulting sampling error will tend to be a little higher than in a pure random sample,[4] but the cost savings may make the trade-off well worth while.

 

Somewhat similar to a cluster sample is a “stratified” sample.  Suppose, for example, that you were studying opinion among students at a university, and wanted to be sure that the numbers of lower division, upper division, and graduate students were representative of the student body as a whole.  You might proceed by dividing the student body into three strata representing these categories, and then select students at random from each of the strata in proportion to their share of the student population.  Sometimes, a research design will call for deliberate oversampling of some small groups so that there are sufficient cases to permit reliable analysis.  (If the university consists mostly of undergraduates, you might need to oversample grad students in order to have enough of them on which to base comparisons with undergrads.)  However, any time analysis is carried out that combines the different strata, cases must we weighted in order to correct for this oversampling.     (In our example, failure to weight cases would result in overestimating the graduate student component of student body opinion.)

 

Many so-called public opinion polls fail to employ random sampling methods.  As a citizen, as well as a student of political science, it is important that you be able to recognize such polls, and their severe limitations.  You may have been stopped at a shopping mall by someone with a clipboard, and asked some questions about your shopping habits.  Maybe while watching a college football game on TV you have been asked to call a toll free (or toll) number to vote for your choice for the Heisman award or, while reading a newspaper online, clicked on a question about policy in the Middle East.  Perhaps you have clipped, filled out, and mailed in a questionnaire in a magazine.   

 

None of these surveys employ anything like random sampling.  Most rely primarily on self-selection, and those who opt to call, click, or mail in a survey may well differ systematically in their views from those who do not.  Even if those questioning customers at the mall are careful to include representative numbers of men and women, older and younger shoppers, people of different races, etc., this approach , called “quota” sampling, should not be confused with stratified random sampling, since there is no guarantee of representativeness within the various groups questions.  Those visiting the mall may or may have different views from demographically similar people who, for example, shop on line or at neighborhood markets.

 

Statistical methods used to make inferences about populations based on samples cannot legitimately be applied to non-probability based samples.  Such samples should be avoided if at all possible.  When you see surveys based on non-probability based samples reported in the media, you may find them interesting or entertaining, but should not take them very seriously.

 

Even in the best designed surveys, strict random sampling is a goal that can almost never be fully achieved under real world conditions, resulting in non-random (or “systematic”) error.  Let us assume that a survey is being conducted by phone, and that interviewers are available who are fluent in all languages spoken by potential respondents.  Not everyone has a phone.  Not all who do are home when called.  Those who are home may refuse to participate, especially if they have had their patience tried by telemarketers.  People who have phones, who are at home when called, and who agree to participate may differ in systematic ways from other potential respondents.   (Surveys done on weekends, for example, tend to include proportionately more Democrats than those done during the week.) 

 

Apart from non-randomness of samples, there are other sources of systematic error in surveys.  Slight differences in question wording may produce large differences in how questions are answered.  The order in which questions are asked may influence responses.  Respondents may lie.

 

Journalists who use polls to measure the “horse race” aspect of a political campaign face particular problems.  One is trying to guess which respondents will actually turn out to vote.  Pollsters have devised various methods for isolating the responses of “likely voters,” but these are basically educated guesses.  Exit polls, in which voters are questioned as they leave the voting area, avoid this problem, but the widespread use of absentee voting in many states creates new problems.  These issues are usually not a problem for academic survey research.  Such surveys are not designed to predict future events, but to analyze existing patterns.  Some such surveys are even conducted after an election is over.  The American National Election Study, for example, includes both pre and post election interviews.  Post election surveys are not without their own pitfalls, however.  Respondents will sometimes have a tendency to report voting for the winner, even when they did not.

 

While surveys can be conducted by mail, these usually yield very low and often unrepresentative response rates, and so the preferred survey methods are face-to-face or via telephone.  The General Social Survey  still employs face-to-face interviews.  The American National Election Study split the 2000 sample between face-to-face and telephone interviews, went to an all-telephone survey in 2002, and plans to return to face-to-face interviews in 2004.

 

In general, however, telephone surveys have increasingly become the method of choice.  The biggest advantage is cost.  The per interview cost of telephone interviews is simply far less than what is required when interviewers are sent door- to-door (thus spending more time getting to interview sites and incurring travel expenses).  There are other factors favoring the use of the telephone.  Interviewers can be more easily trained and more closely supervised.  Problems that arise can be dealt with on the spot.  CATI (Computer Assisted Telephone Interviewing) technology can be employed for such things as random-digit dialing, call-backs, and data entry.

 

A disadvantage of telephone surveys compared to door-to door, face-to face interviews is their relatively low response rates.  The response rate for the study of the Political Personality of America’s College Students, a telephone survey, had a response rate of only 18.7 percent[5].  When the American National Election Study split its sample between face to face and telephone interviews for its 2000 pre-election survey, it obtained a response rate of 64.8 percent for the former, compared to 57.2 percent for the latter[6] .   An analysis of a number of telephone and face-to-face surveys showed that the latter were generally more representative of the demographic characteristics of the general population[7] .  

 

When samples can be compared to some known characteristics of the population, and samples can be weighted  to compensate for under or over representation of some segments of the population.  There is often no way of knowing, however, whether respondents and non-respondents differ in their political attitudes and behavior. 

 

 

Optional Exercises

 

1.         Assume that you wish to survey students at your college or university regarding their opinions on various issues, their political party loyalties, and their voting intentions in the next election.  Design an appropriate questionnaire, decide how many students you will need for your sample, and spell out how the sampling will be done.  (If you are actually planning to carry out such a survey, be aware that your institution has, or should have, rigorous legal and ethical standards for conducting research involving human subjects.  Allow plenty of time to find out what these standards are, and be sure to incorporate them into your research design.)

 

2.         Visit http://www.learner.org/exhibits/statistics/.  In what ways does the survey described here reflect good sampling and questionnaire design?  In what ways does it not?

 

For Further Study

 

Survey Research:

Nelson, Elizabeth N., and Edward E. Nelson, “Survey Research Design and Quantitative Methods of Analysis for Cross-Sectional Data,” California Opinions on Women's Issues -- 1985-1995.  http://www.csub.edu/ssric-trd/modules/cowi/3.htm.

Palmquist, Ruth A., “Survey Methods,” http://www.gslis.utexas.edu/~palmquis/courses/survey.html. 

 

Research Randomizer, http://randomizer.org.

 

The Roper Center, "Polling 101," http://www.ropercenter.uconn.edu/pom/polling101.html.

 

There are several good sources of polling data available online, including PollingReport.com (http://www.pollingreport.com).

 

---