Cell Phones and Political Polls

The most common kinds of public opinion polls long have been conducted by calling a random sample of residential phones. This was OK when nearly every home had a phone, but in recent years a growing number of people, mostly young adults, have decided to use only a cell phone and do without a separate landline in their home.

It's possible to include cell phones in a poll sample, but it's expensive, difficult and seldom done. That means a growing number of cell-phone-only persons are generally not included, and their opinions are not reflected in the results we commonly see published.

How big a problem is this? A study published in January by the Pew Research Center for the People & the Press concluded that results of a poll including cell-only respondents were "virtually identical" to those based only on calls made to landlines. That's reassuring, but we remain skeptical.

Pew commissioned two polls, one in October and one in December of 2007, which together included 2,596 interviews conducted by calls to landlines and 841 interviews conducted by calls to cell phones, using a sample drawn from a national database of cell phone numbers. And of those reached by cell phone, 312 people said their mobile phone is the only one they use.

The researchers reported little difference between the results from the landline-only sample and the larger group of both landline and cell phone users:

Pew: When data from both samples are combined and weighted to match the U.S. population on key demographic measures, the results are virtually identical to those from the landline survey alone. Across more than 100 political and attitudinal questions on the surveys, including cell phone interviews does not change the results by more than two percentage points in the vast majority of comparisons, and in only one comparison is the difference as large as 4 points.

In particular, there is no evidence that the polling in the Democratic and Republican nomination contests is biased by the fact that most polls rely only on landline interviews.

Reasons to be Skeptical

On some questions the differences in opinions were striking. The authors reported that cell-only respondents were 14 percentage points less likely to say Social Security would be important to their vote and somewhat more likely to say immigration would be important, for example.

Poll-takers worry about what such differences might do to the accuracy of their results, and to public confidence in polling generally. Last year Public Opinion Quarterly devoted an entire special issue to the subject of cell-only users and what to do about them. One article predicted that by the end of 2009 more than 40 percent of adults in the United States under 30 years of age will have adopted a "cell phone only" lifestyle. Another found that telephone surveys using only landline calls underestimate the prevalence of health behaviors such as binge drinking, smoking and HIV testing. A wrap-up article stated that "the possibility exists for very sizable coverage errors associated with young adults in future U.S. general population telephone surveys that do not include the sampling of cell phone numbers."

There were already plenty of reasons to treat polling results with caution. To cite only the most recent example, poll-takers are still trying to figure out what went wrong with surveys showing Democratic presidential candidate Barack Obama with a big lead just before the Jan. 9 New Hampshire primary, which he lost by 2 percentage points.

Even assuming that poll-takers could achieve a perfectly unbiased, random sample of the entire population, there is inevitably a statistical margin of error. That means that the laws of chance dictate that about 95 percent of the time the results from the sample may differ by, say, 2 or 3 percentage points from the result that would be obtained if everybody in the population were surveyed. By the same token, however, one out of 20 such polls will produce results that are off by even more than the margin of error.

Furthermore, telephone polling has always missed a fair number of persons who don't have a phone. As recently as 1990, for example, the Census Bureau reported that 5.2 percent of the homes in the U.S. had no telephone, and in several states the figure was 10 percent or more. (In 2000, the Census Bureau said the number without phones had dropped to 2.4 percent but added, "Increased cell phone usage probably played a major role in this dramatic change.") In addition, poll-takers fret about an increasing number of persons who answer their phones but simply refuse to answer their questions.

Poll-takers apply various statistical manipulations to their survey results in an attempt to compensate for those who are missed in phone surveys. That task is becoming more difficult as more Americans do without hard-wired phones, giving the wise citizen one more reason not to put too much weight on any particular finding from a telephone poll.

–Brooks Jackson