« Yikes! I Missed It | Main | Escape From Springfield »

March 5, 2008

Reporting the (less than) Facts

Jay Rosen posted an interesting assignment in "pattern recognition" for his readers last week. This week he considers the results. All of it is worth your time. Unfortunately, I was just too busy with other things to give the assignment the time and attention it deserved. You'll see from the following, however, that I have far too much time on my hands this week :-)

In his essay about the results of the assignment, Rosen praises the Washington Post for its feature called The Fact Checker. I agree. This is good stuff--exactly the kind of thing I like to see political reporters do. The basic idea is to vet the candidates' assertions of fact and award "Pinocchios" for those assertions found wanting. The scale is 1 to 4, and you will find the criteria here.

Rosen, however, laments that the feature does not have a chart comparing candidates and their Pinocchios. He wonders why. I have a guess, and it has to do with the strange way journalists understand objectivity. Quickly, because I have another agenda here: Journalism is uncomfortable asserting facts that appear to make evaluations. Senator Blowhard says X (a fact that he said it) and Senator Numnutz counters with Y (a fact that he said it). To fact-check these politicians (beyond the fact that they said something) is to be unfair by the strange alchemy of modern political reporting. Readers are supposed to figure it out. So the Post is already doing a transgressive thing with The Fact Checker.

Now they need to take the next step toward true objectivity and tell us how the candidates compare. If you have good criteria applied objectively (as academics understand this term), then the results may certainly favor some candidates over others, but the results just as certainly will be fair and objective.

I've decided to do (some of) the work the Post should do. This is a quick-and-dirty, raw analysis. Don't be too quick to jump to conclusions. Part of what I'm doing here is demonstrating how difficult it can be to tell us how the candidates compare (and, thus, I'm arguing for more analytical training for journalists).

Since sometime last October, Hillary Clinton has earned 25 total Pinocchios, Barack Obama has earned 28 total Pinocchios, and John McCain has earned 11 total Pinocchios. These data, however, don't tell us much because there is a scale of 1 to 4. But one might sense that McCain, overall, has his facts straight more often than Obama and Clinton. But I'm seeing a problem. If you use the site's search function to pull up everything on the candidates (a drop-down box with pre-sorted results for each candidate), what you notice is there has been far more coverage of Obama and Clinton. The search results return 40 items for Obama, 43 items for Clinton, and 19 for McCain. What's up with that? (I suspect it may be because last October many political analysts thought the McCain campaign was sunk.)

Here's the raw data for each candidate, i.e. what each candidate was awarded:

Obama (14 assertions challenged):

1 Pinocchio = 4
2 Pinocchios = 7
3 Pinocchios = 2
4 Pinocchios = 1

Clinton (15 assertions challenged):
1 Pinocchio = 5
2 Pinocchios = 10
3 Pinocchios = 0
4 Pinocchios = 0

John McCain (4 assertions challenged):
1 Pinocchio = 0
2 Pinocchios = 1
3 Pinocchios = 3
4 Pinocchios = 0

How I got these data: I used the drop-down search function and searched each entry for the Pinocchio logo. I read the concluding analyses to make sure I was counting the Pinocchios for the proper candidates. I ignored all entries that gave results other than Pinocchios (i.e. I saw no logo).

Are these data interesting? In a political/rhetorical sense, they might be because one could easily (mis)use (intentionally?) these data to create arguments for or against any of these candidates. For example: John McCain doesn't lie often, but when he does they are whoppers. Or: Clinton lies more than Obama, but Obama's lies are bigger. Those assertions are obvious nonsense. Perhaps the Post fact checkers think they might slip into nonsense if they attempt to compile and compare the data. Self-knowledge is a good thing. Journalists should have a firm grasp on their limitations.

What the Post needs to do, however, is either find a reporter who knows something about statistical analysis or contract a local egghead to do the work. My guess is some professor would jump at the opportunity.

Wouldn't it be nice if journalists could handle such analysis on their own? We might then hasten the day when a mistaken sense of objectivity no longer leads to unfair fairness.



Posted by acline at March 5, 2008 12:28 PM | | Spotlight