: What about the issues?…The problem with polls…
Richard Morin runs down some of the problems with modern polling:
Cell phones, Caller ID and increasingly elaborate call screening technologies make it harder than ever to reach a random sample of Americans. Prompted by the popularity of do-not-call lists, a few state legislatures are considering laws that would lump pollsters in with telemarketers and bar them from calling people at home.
Costs are soaring as cooperation rates remain at or near record lows. In some surveys, less than one in five calls produces a completed interview — raising doubts whether such polls accurately reflect the views of the public or merely report the opinions of stay-at-home Americans who are too bored, too infirm or too lonely to hang up.
But there’s a bigger problem: the reporting of polling by journalists and bloggers. All too often the reporting fails to accurately describe the margins of error or the questions asked. Without these adequately explained, any report of a poll is pure (and dangerous) nonsense.
For example, suppose you have a presidential preference poll that puts John Kerry’s support at 48% and George Bush’s support at 46%. Does this mean Kerry leads? If you believe so much political reporting and blogging it does. But check the margin of error, if given, and you’ll likely find that this is a tie. In a poll with a typical margin of error of 3.5 points, anything within the margin is a tie. And any claim of a lead is bunk.
Another example: Suppose you encounter a poll in which one of the candidates leads outside the margin of error. Is this significant? Maybe. Now you have to know what question was asked and how it was asked and to whom it was asked. If you don’t know these things, then you still don’t know what the poll means.
But the poll will have meaning. The media (mainstream and otherwise) will give it meaning. And if people act in political ways based on the meaning given, then that’s what the poll means. Get my meaning?
Do polls as reported in the press affect how people vote? Hmmmmm…the problem is obvious.
UPDATE (2:15 p.m.): William Saletan offers a useful consumer’s guide to the polls:
Why do the polls disagree so much? And which ones should you buy?
That would be easier to decide if you could buy polls the way you buy canned food or cereal, with all the ingredients disclosed on a label. But pollsters don’t package their products that way. Consequently, most of us believe, mistakenly, that a poll is a simple tabulation of a random sample of voters. In reality, polls are full of additives and preservatives, subtractions and selective multiplications, none of which are generally published. The reason many polls consistently differ from others is that their hidden ingredients differ.
It seems to me that journalists should not report on polls unless they understand and present the hidden ingredients.
This is another example of journalists not be willing (or able) to explain the complexities of the situation. Unless you dig deeper, you won’t understand how these things work. Ironically, I think this has two contradictory effects: on the one hand, people think that polls are meaningless because they don’t understand the science they are based on and at the same time they put too much weight on polls (at least ones supporting their position) because they don’t understand the limitations. I think you see a lot of this among journalists as well, at least among people writing for opinion journals, such as The New Republic. I have seen stories that quote polls uncritically that support their story and attack the underlying basis of polls that don’t.