Needs more than an academic ethos…
As I suspected, the Government: In and Out of the News study, conducted by The Center for Media and Public Affairs and funded by The Pew Charitable Trusts, was short on theory and long on content analysis.
Today, I’m going to discuss problems with the study. Later this week, I’ll discuss what I believe are its useful contributions to understanding the relationship between the press and politics.
Here is the first paragraph of the conclusion:
This study encompassed nearly 30,000 news stories from nine national and local media outlets over a span of two decades. Its unique breadth and depth permitted us to test some widely held assumptions about news coverage of the federal government. In some respects the results confirmed or built upon the findings of previous researchers. In others, however, they challenged misconceptions that are held by scholars, politicians, and journalists.
The first sentence is entirely correct. The following sentences are misleading.
The ethos of the report is academic, but its substance is unchallangeable because we are given no idea how this report was conducted. We know what time periods and news outlets were chosen for study. And we know a little bit about what articles or news broadcasts were chosen. But we are given no clue about the substance of the content analysis. The report’s methodology section merely re-states much of the PR puffery from the Center for Media and Public Affairs website. This tells us nothing other than the Center is mighty proud of its methods:
Our analysis of media coverage of the U.S. federal government relied on the social science method of content analysis. Content analysis is a technique which allows researchers to classify statements objectively and systematically according to explicit rules and clear criteria. The goal is to produce valid measures of program content, and the hallmark of success lies in reliability. Other investigators who apply similar procedures to the same material should obtain similar results, although their interpretations of those results may differ. Clear rules and standards have to be set for identifying, measuring, and classifying each news story. In making each decision, researchers are applying these rules, not expressing their own opinions. If the rules are sufficiently clear, two investigators working independently will come to similar conclusions, regardless of their personal opinions about the subject matter. (148)
This says: Trust us! But I’d rather not.
If other investigators want to apply similar procedures to check this data, they’ll have to contact the Center for Media and Public Affairs to get the information that should have been included with the report (if it were a proper academic study). I sent this e-mail message to the Center yesterday:
To whom it may concern:
I have two questions regarding your recent report entitled Government: In and Out of the News.
1. While your report and web site offer an evaluation of your methodology, you do not explain it. So your results cannot be independently checked and your research cannot be reproduced. May I see your methodology and research procedures? Please send them by e-mail. Or you may send them by U.S. mail to the address in the signature below.
2. You twice cite (Robinson and Sheehan 1983) in the report on pages 26 and 137. No such source exists in your references. Would you please send the proper citation by e-mail?
Thank you for your help.
Oh, yeah, that second point: S L O P P Y
The first reference to (Robinson and Sheehan 1983) sent me scrambling to the references because it apparently supported this dubious point:
Comments from nonpartisan sources are more influential, since viewers are more likely to discount explicitly partisan evaluations. Therefore, media scholars sometimes base their measures of bias on comments by sources who are not identified with either party.
There are two fat “sez-whos” here and no way to check them out.
It’s difficult to criticize the methodology of this report because I don’t know what it is. But I know that the data suggest some interesting things about how press coverage of politics has changed since 1981. I’d really like to be able to evaluate what I’m seeing.
Here area few observations based on my gut reaction. I may amend these after I am able to consider the methodology.
1. Confining the analysis to front-page articles unnecessarily restricts the data and fails to take into account changes in newspaper design and content since 1981 (USA Today debuted in 1982). Most newspapers use a greater percentage of the front page to run features, local news, and entertainment promos. Photos are bigger now. And there are more graphics of all kinds. This shrinks the news hole and forces articles that may have been given front-page play in 1981 to the inside pages.
2. I can think of no justification for not including articles and broadcasts from the first Bush administration.
3. The concept of “tone” is flawed. The researchers apparently coded quoted statements positive or negative and used this coding to determine a score of some kind. In many cases, you get data that look like this: one third positive or one third negative. No kidding? Here’s how that happens: Reporter interviews 3 sources regarding a political situation A. Politician X is positive. Politician Y is negative. Expert Z is either positive or negative (and usually negative, I would argue, because of the bad news and narrative biases). Further, a truth statement in regard to political situation A may exist (and good reporting should discover it). The concept of tone does not take this bit of important context into consideration.
The “tone” concept causes other strange problems. For example, from page 22:
Conversely, comments directed toward presidential appointees and executive agencies became more negative after the attacks, dropping to 31 percent positive after 9/11 from 48 percent positive previously earlier. This included pointed questioning as to why the nation’s intelligence services did not do more to protect the public against the attacks, as well as the Justice Department’s secretive and sometimes controversial detention procedures regarding possible terrorists.
The concept of “tone” in this report encodes statements as negative even if they are an important part of doing good journalism, e.g. asking tough questions. Would the “researchers” be happy if reporters only talk to sources who are positive about everything?
They make the claim that journalism is negative in tone. Sounds bad. But it appears to me they are improperly encoding skepticism and disagreement (because content analysis such as this fails to account for the role of rhetoric). While the bad news and narrative biases create some civic problems (even real instances of political and partisan bias), proper skepticism is an aid to making civic decisions. Something valuable I take from the data of this report: Perhaps a two-thirds negative tone is a good thing. I’ll have more to say about this in my (second) final installment.
4. Simple percentages do not make a sophisticated statistical analysis. Something is 42% and its opposite is 58%. Is that difference important? Is it significant? Counting up this and that is just the first step. Proper statistical analysis is necessary to evaluate the data in this report. I’d rather not take their word for it. And I’d rather be able to reproduce their research for myself. (Disclaimer: I do no do statistical analysis. I do, however, incorporate the analyses of others in some areas of my work.)
5. What the heck is a “discussion”? They use the word in two different ways. In the TV section it appears to refer to news segments in general. But in the national newspaper section, a “discussion” is this odd beast:
For greater precision, we based the bulk of our analysis not on news stories as a whole but on the individual discussions of government that appeared within each article. This allowed us to differentiate discussions of the three branches of government and their component parts, even when they were discussed within the same story. To qualify, an individual, group, issue, office, or organization related to the federal government had to be discussed for at least two full paragraphs within an article.
So we’re going to arbitrarily decide to study units of at least two paragraphs and strip them of their context and suppose we’ll be able to say something useful? Hmmmmmm… Are these guys aware that the concept of paragraphing in journalism is largely arbitrary? Sentences, which are often single paragraphs in news writing, when written well are designed to communicate single (important) ideas (remember that inverted pyramid structure stuff?). Why exclude a unit of importance recognized by professional practice?
That’s enough for now. I’ll summarize this way: This is a sloppy, opaque report. Its ethos says “academic,” but that’s just a veneer. (Note: My evaluation may change after I have a chance to see the methodology.)
I haven’t read the whole report (don’t know if I will), but the methodology is spelled out better in the paper’s introduction (pages vi to viii).
Josh… That section gives a general description of the method. But there is nowhere near enough information about how this study was conducted. For example: What was the coding procedure? It’s not possible to evaluate this data without knowing that…and a few other things, too.
Here’s what they’ll provide for the reference (my bet):
And you’re correct – it was SLOPPY to omit that reference in the report.
It will be interesting to hear what response you get from them on methodology (and perhaps data).
The front page evolution is an interesting insight that did not occur to me when reading the report. However, isn’t there another side to that coin? If the newshole on the front page is smaller, are the reading habits such that there isn’t a corresponding loss of signal to the reader?
Less news on the front page means less news read by the consumer?
Your discussion on tone is interesting, and certainly your academic strength. But I’m confused by it. You explain that finding a negative tone is unsurprising because of the narrative and bad news bias – and that those biases should be included as context in measuring tone?
It would be more informative to divide tone into categories with greater fidelity than just positive and negative. Constructive, suspicious, accusatory, Gotcha!, etc., might be some.
I’m not comfortable with the “tough” question category because it’s too often a cop-out … i.e., from PressThink:
re: “If the newshole on the front page is smaller, are the reading habits such that there isn’t a corresponding loss of signal to the reader?”
[my comment edited to eliminate stupid answer]
I think USA Today was the worst thing to happen to print journalism in the 20th C. What a newspaper puts out front speaks volumes about its seriousness. I don’t know about a corresponding habit to compensate. I hope so. I suspect not.
re: “Less news on the front page means less news read by the consumer?”
I don’t know. I would hope not. Finding out would make a good study.
re: “Your discussion on tone is interesting, and certainly your academic strength. But I’m confused by it. You explain that finding a negative tone is unsurprising because of the narrative and bad news bias – and that those biases should be included as context in measuring tone?”
Negative tone in an of itself may be good or bad. How can we know until we know the context? But I find this a fascinating moment in the study (I learned something, maybe): What if a negative tone is structural in exactly the way I suggest. Perhaps we need to be asking questions about what’s causing the negative tone structurally and whether such a tone is even a problem. The data suggest to me that a negative tone is natural to the practice of journalism. Now, don’t read too much into that. Because we still need to determine the value of tone. But I find this a very interesting proposition to consider.
re: “It would be more informative to divide tone into categories with greater fidelity than just positive and negative. Constructive, suspicious, accusatory, Gotcha!, etc., might be some.”
That could be interesting–especially since such categories would require understanding individual statements in context.
re: “I’m not comfortable with the “tough” question category because it’s too often a cop-out”
Yes, it certainly can be. But not all tough questions are “gotcha” questions–although POV plays a big role there
All the more reason to be sure we keep all communicative units of study in context.
“What if a negative tone is structural in exactly the way I suggest.”
Let’s assume it is. Then what would account for the difference in negative tone between Reagan, Clinton and Bush? Or between Democrats and Republicans in Congress.
Would the structure cause such differences, or wouldn’t a “structure” suggest consistancy?
re: “Would the structure cause such differences, or wouldn’t a “structure” suggest consistency?”
There are a few things we still need to know, but I would assume that any structure would show differences of some kind. And as far as consistency goes, this negative tone seems very consistent given the data. What we don’t know, because these guys apparently didn’t do the math, is whether the differences between Clinton and the others are statistically significant.
It would help if we had these two things:
1. Similar data on all post-war presidents.
2. A proper statistical analysis of the data.
3. Analysis of a second year.
Two … three … who’s counting?
I was thinking about table 78 & 79 starting on page 139?
I also should tell you that I’m intrigued by the choices you’re making in argument vs. point analysis of this study.
re: who’s counting
Apparently not me
re: tables 78 and 79
What they are claiming may very well be true (and interesting). But I have two questions:
1. How did they decide to set an acceptable level of significance at .05?
2. What’s the standard deviation?
They could be onto something really important, but I’d prefer not to just take their word for it.
re: argument versus point
“This is a very common experience for me. I often ignore or disagree with examples in favor of dealing directly with points.”
(Which I kinda did in the prequel.)
Often, but not always. In this case we’re dealing with specific research. I’d like to know that it is well done. And I’d like to know how it was done.
“We shouldn’t confuse “point” and “argument.” If I disagree with the examples, I am disagreeing with the argument. I may still agree with the point. I may have other ways of arriving at there. Perhaps the question should read this way: If one disagrees with examples, should one disagree with the argument? Yes.”
I cannot make a decision yet on the argument because I still have questions about it. Regarding the last assertion, so far I have problems with the examples (specific research) so I’m skeptical about the argument.