No Cellphones, No Callbacks, No Neutrality

We've hoped for a while that people have learned, through hearing about polls, that one poll by itself doesn't constitute a game-changer. It should only be regarded as part of a larger trend--because absent a major, substantive scandal or the death of one candidate, polling as a whole rarely swings suddenly in a new direction. However, even in this context the latest New England College poll does give us something to think about: is Sen. Jeanne Shaheen really down against Scott Brown?

In its survey of 921 New Hampshire likely voters, New England College found that Sen. Shaheen was just barely behind the former Massachusetts senator by less than one percentage point, well within the margin of error. But then again, a month ago Sen. Shaheen routinely led by 6 or 7 points. "Within the margin of error" is not a phrase she wanted to hear. And it's not as though this poll is a truly isolated incident. Recent polls from UMass - Amherst, the University of New Hampshire, and SurveyUSA all show a close race. So should we accept this New England College survey as part of the story? 

I highly doubt it. Or at least, it shouldn't be held as part of the story on an equal level with the other pollsters. The major problem with the New England College poll is that the number of days over which it was conducted--what I call the "date spread" in my spreadsheets--is one. All interviews were conducted on one day, October 16. 

This would not be a problem in itself, but what it implies is that the College is not calling back respondents who don't answer the phone. There's simply not enough time to call back respondents you didn't reach the first time. When I worked the phones for the Rockefeller Center's State of the State poll in April, I sat in a cramped, damp room with 20 to 30 undergraduates from six to nine in the evening for four days in a row. We ended up with a sample size of 412 registered voters, and we rarely got a majority of respondents to answer the phone on the first or second try, not to mention the thinly-veiled disdain from voters who hung up within ten seconds, or the awkward moments when we were told by the target voter's spouse that the voter in question had actually died several months ago.

But I digress. The point is that New England College must be cutting some serious corners: with a student body one fourth the size of Dartmouth College (the parent institution of the Rockefeller Center) and in a quarter of the time, New England College somehow got a sample size more than twice as large. The most likely explanations for this is that New England College has identical problems to that of Rasmussen Reports in the 2008 and 2010 elections. Like New England College, Rasmussen released an unprecedentedly large number of polls with fairly large (around the 750 LV range) sample sizes. Rasmussen's results were significantly more biased toward Republicans than those of other pollsters. But it's not as simple as "Rasmussen was biased toward Republicans". A number of specific problems with Rasmussen's methodology created a Republican-leaning house effect:

• Robo-calling. Rasmussen used automated telephone interviews to conduct its surveys. This wouldn't be a problem in itself, but it's actually illegal for robocallers to dial cellphones. Rasmussen got around this legal obstacle by just not calling cellphones. Since about a third of American households have no land line--and that third tends to be younger and thus more liberal--Rasmussen missed out on an important, non-random voting bloc.
• Single-day polling. Rasmussen's surveys were conducted within a single four-hour window on a single day. In doing this they missed out on individuals who worked late on weekdays and oversampled senior citizens: as one of my supervisors for the State of the State poll said, "old people have nothing better to do than to wait for someone to call them."
• Untargeted interviews. Instead of calling a household and asking for Mister Jane Doe, Rasmussen would call the household of Mister Jane Doe and interview the first registered voter to pick up, which could very well have been Mrs. Jane Doe or Mister Jane Doe, Sr. (That's Mister Jane Doe's father, while we're hypothesizing about making names unisex.) This biases the sample toward the type of person who would be first to the phone, whoever that might be. 

I'm not sure about the last one, but the New England College survey definitely suffers from the first two problems. Interviews were all conducted on the same Thursday, and the results "were obtained through automated telephone interviews". As a result, it misses out on the same groups Rasmussen did.

Speaking of which, however, Rasmussen does not appear to have the strongly Republican house effect it displayed in earlier years. This may be because they've fixed the second problem--they've increased their date spread, which allows them to call back those households which don't respond at first and to target their respondents more carefully (although it doesn't solve the cellphone barrier, which they never will be able to unless they either conduct live interviews or the law changes). In fact, New England College itself only recently started showing signs of a Republican house effect. A little over a month ago they released a survey of 1,331 likely voters showing Sen. Shaheen up 50-43 over Brown--pretty much exactly in line with the average. Incidentally, that was the last time New England College conducted a survey with a date spread of more than one; since then it has only released single-day polls, and the results have been substantially more Republican-leaning than the rest of the polling would suggest.

We do penalize this in the average. First, a simple corrective anti-house effect is applied to the New England College. This both deweights the poll slightly in the poll and brings it closer (although not entirely) in line with other polls. However, this is a fairly light correction, since we tend to give pollsters the benefit of the doubt. The next penalty is directly applied to the weight of the poll: we deweight single-day surveys by a factor of ten. This penalty is quite drastic: even much older polls, like a University of New Hampshire poll from three weeks ago, carry more weight than the New England College poll from a week ago simply because the UNH methodology implied better practices than the New England College poll did. "Garbage in, garbage out," they say: but by doing this we hope at least to dilute any potential garbage. In the scheme of things, when we've got a bunch of polls out, it has very little effect anyway.