Granite Geek: The Science of Telephone Polling

Oct 20, 2015

This far into the campaign season, polls are generating lots of headlines. And if you live in New Hampshire, polling firms have likely been calling you and hundreds of other Granite Staters. But how do those polling firms find you? How do they choose their questions, and what do they do with your information?  For more on this, we turn to David Brooks who’s a reporter with The Concord Monitor, writer at GraniteGeek.org, and he’s moderating a Science Café panel discussion about this very subject Wednesday, October 21st at 6 p.m. at Killarney’s Pub in Nashua.

We’re going to leave out any discussion of any recent polls about the current candidates running for president.

I’m sorry there are candidates running for president?  I wasn’t aware of that.

Can you believe it?

I didn’t know that.

We’re going to ignore them and focus on the science of polling. So, my first question is how did they get my phone number?

Well, they got your phone number by random digit dialing, which is, in other words, basically they dial every possible combination of numbers, sort of. And actually, as part of preparing for the Science Café tomorrow night, I learned that random-digit dialing is more interesting than I thought. So, under the basic various federal laws, in particular random-digit dialing cannot be done by robocalls, by these automated callers. And that’s why you don’t, or at least you shouldn’t get any robocalls to your cell phone, because there’s no directory for cell phones.  So robocalls can only use directory numbers like the white pages which means it only comes to landlines, for those of us who still have landlines. On the other hand, if your poll is done by human beings sitting in a telephone bank, then you can use random digit dialing. So they are the ones that come to your cell phone.

But I imagine polls with a real human being are way more expensive than the robocalling.

A lot more expensive. I can’t say exactly.  I talked to Zachary Azem at the UNH Survey Center, who’s going to be one of the panelists, and he declined to give me the actual number because, I mean, that’s one of the things they do is they sell these polls to companies. But looking around it seems that to get a poll that has 500 responses, which is what you usually need, it’s going to cost $10,000 to $20,000, whereas a robocall is maybe a quarter or that, a tenth of that, even.

How do these polling firms go about crafting the questions?  Is there a way to describe the perfect poll question that they come up with?

Well, so the way it was described to me is the first thing you really need to do is figure out your target audience. Who do you want to get information from? So for example, for these political polls you’ve got to be a registered New Hampshire voter who’s likely to vote. So that’s the single most important thing a poll has to do. A poll that is talking about buying cigars and interviews 10,000 people, 9,000 of whom  don’t smoke, it’s a useless poll.

The actual wording of the questions is a least as much art as science. There is considerable literature about just changing the order of the words, the order in which you ask questions, the order of words within a question, the tone of voice can significantly alter the response you get.

The example I think is most interesting at the moment with political polls is the standard question you get-- if the election were held tomorrow, who would you vote for?  That seems like a very intelligent way to ask, but there is a mounting body of evidence that you can get a better reflection of what the voting population actually thinks if instead you ask people, if the election were held tomorrow, who do you think would win? Because the second question, you’re not asking for my personal opinion about elections, you’re asking for what I think the society around me is doing. And if you get lots of people saying what they think society around them is doing it seems to be a more accurate depiction of what society around them actually is doing.

And once these polling firms have gathered their data, made these thousands of phone calls, what do they do with that data?

They crunch it numerically. There are some very well established mathematical algorithms that tell you how confident you can be. So the general thing is a 95 percent confidence interval and say a margin of error of 4 percent. So you’ll say you know who Jones is. You know 30 percent of the voters like Jones and 20 percent of voters like Smith with a margin of error of 4 percent,  and since the difference between them is wide enough, you’re very confident that Jones would win 95 percent of the time. So if 20 identical elections were held in 20 parallel universes, Jones would win 19 of them, that’s what that poll says. On the other hand, if Jones gets 20 percent and Smith gets 25 percent and there’s a margin of error of 4 percent, that seems like it still says the one guy is winning, but actually it isn’t because the margins of error overlap. So there’s some mathematical intricacies there that are fun to play with.

So David, on a scale of one to five, five being totally awesome, one being just terrible, how would you rate this conversation?

Totally awesome, as always.