According to a new study buzzing through the media, no amount of alcohol is safe when it comes to driving. “Just one beer,” says Time magazine, is associated with “incapacitating injury and death.” This alarming research, courtesy of the latest issue of Addiction, has obvious policy implications, which study co-author David Phillips, a University of California San Diego sociologist, spelled out in a widely quoted statement: “We hope that our study might influence not only U.S. legislators but also foreign legislators in providing empirical evidence for lowering the legal blood alcohol content, or BAC, even more. Doing so is very likely to reduce incapacitating injuries and to save lives.”
But if you look closely at the paper, there are other, rather more surprising, take-home messages. If you live in the West, for instance, going from zero to one drink is associated with a lower risk of severe injury. And, as my colleague S. Robert Lichter at George Mason University’s STATS points out, the published data actually suggests that the overall relative severity of injuries drops the more alcohol is consumed, at least up to a point. As he notes: “Following the authors’ policy inferences from their findings, one might advise drivers to avoid that first drink, but if they are already up to 0.02 percent BAC, they should take another drink to bring them up to 0.03 percent or 0.04 percent BAC, since that would reduce their likelihood of serious injury. Of course, no one would seriously make this suggestion.”
How can you reconcile data suggesting that one drink impairs your driving skills but two drinks doesn’t? It doesn’t make sense — and that calls into question whether the researchers made significant errors in the way they assembled and analyzed their data.
This isn’t the first time the media have reported fascinating findings that turn out to be confusing — and considerably less newsworthy — under closer scrutiny. The question is why does this keep happening? One answer might be that science and journalism focus on different kinds of error.
Consider the first systematic study to look at how the media covers science — “News Source Perceptions of Accuracy of Science Coverage,” by James Tankard and Michael Ryan — which appeared in Journalism Quarterly in 1974.
Just 8.8 percent of science stories were judged error-free by a panel of scientists, compared with 40 percent to 59 percent for other beats, and Tankard and Ryan identified a staggering 42 types of error, with a lack of methodological detail and a failure to place the new research in the context of existing research as the two most serious failings.
The authors admitted that their finding of an average of 6.2 errors per story might have been derived from the sheer number of possible errors, and a follow-up study, literally called “Follow-up Study of Science News Accuracy,” in the same journal two years later found that when the kinds of error were reduced to 11, 29 percent of stories were error-free and that the error rate declined to 2.16 per story.
Even if you accept the most charitable accounting, these findings make even grimmer reading now than they did in the 1970s, given that dedicated science journalists have become an endangered species in newsrooms, and so much reporting on new research goes little further than the press release accompanying the study (another source of error, according to a 2009 study in the Annals of Internal Medicine).
But these accuracy studies were not without their own controversy; namely, what exactly was considered an error? Journalists tended to acknowledge objective errors, such as clear and simple misstatements of fact or misspellings, while the scientists focused on substantive errors that were interpretative and required analyzing the data and knowing the full weight of the evidence.
Journalism just doesn’t feel comfortable with interpreting news or reporting or analyzing numbers, so the criteria for accurate science reporting have become limited to those facts that were clearly objective and understandable — did I quote Professor X accurately, get his title right, correctly report what his research says?
Which means we end up with the perplexing situation that, even though there is nothing inaccurate with the media coverage of the addiction study, we’d be wrong to accept the coverage — or the authors’ policy recommendations — at face value.
Of course, the other critical factor is whether we do, collectively, accept the coverage at face value. The evidence, from a 1990 study in Health Education Quarterly — “Evaluating Understanding of Popular Press Reports of Health Research” — is not comforting. The overall rate of college-age reader misunderstanding was 40 percent, which suggests that the public is as likely to misinterpret accurate reporting as inaccurate reporting.
So, to sum up: You have the constant publication of new research, which may or may not be flawed (and distorted by its own PR) or incompletely reported by the media to the point of misrepresentation, which is then likely to be misunderstood by a significant amount of the public. The rational response is not just cui bono, but why bother?
But if you look closely at the paper, there are other, rather more surprising, take-home messages. If you live in the West, for instance, going from zero to one drink is associated with a lower risk of severe injury. And, as my colleague S. Robert Lichter at George Mason University’s STATS points out, the published data actually suggests that the overall relative severity of injuries drops the more alcohol is consumed, at least up to a point. As he notes: “Following the authors’ policy inferences from their findings, one might advise drivers to avoid that first drink, but if they are already up to 0.02 percent BAC, they should take another drink to bring them up to 0.03 percent or 0.04 percent BAC, since that would reduce their likelihood of serious injury. Of course, no one would seriously make this suggestion.”
How can you reconcile data suggesting that one drink impairs your driving skills but two drinks doesn’t? It doesn’t make sense — and that calls into question whether the researchers made significant errors in the way they assembled and analyzed their data.
This isn’t the first time the media have reported fascinating findings that turn out to be confusing — and considerably less newsworthy — under closer scrutiny. The question is why does this keep happening? One answer might be that science and journalism focus on different kinds of error.
Consider the first systematic study to look at how the media covers science — “News Source Perceptions of Accuracy of Science Coverage,” by James Tankard and Michael Ryan — which appeared in Journalism Quarterly in 1974.
Just 8.8 percent of science stories were judged error-free by a panel of scientists, compared with 40 percent to 59 percent for other beats, and Tankard and Ryan identified a staggering 42 types of error, with a lack of methodological detail and a failure to place the new research in the context of existing research as the two most serious failings.
The authors admitted that their finding of an average of 6.2 errors per story might have been derived from the sheer number of possible errors, and a follow-up study, literally called “Follow-up Study of Science News Accuracy,” in the same journal two years later found that when the kinds of error were reduced to 11, 29 percent of stories were error-free and that the error rate declined to 2.16 per story.
Even if you accept the most charitable accounting, these findings make even grimmer reading now than they did in the 1970s, given that dedicated science journalists have become an endangered species in newsrooms, and so much reporting on new research goes little further than the press release accompanying the study (another source of error, according to a 2009 study in the Annals of Internal Medicine).
But these accuracy studies were not without their own controversy; namely, what exactly was considered an error? Journalists tended to acknowledge objective errors, such as clear and simple misstatements of fact or misspellings, while the scientists focused on substantive errors that were interpretative and required analyzing the data and knowing the full weight of the evidence.
Journalism just doesn’t feel comfortable with interpreting news or reporting or analyzing numbers, so the criteria for accurate science reporting have become limited to those facts that were clearly objective and understandable — did I quote Professor X accurately, get his title right, correctly report what his research says?
Which means we end up with the perplexing situation that, even though there is nothing inaccurate with the media coverage of the addiction study, we’d be wrong to accept the coverage — or the authors’ policy recommendations — at face value.
Of course, the other critical factor is whether we do, collectively, accept the coverage at face value. The evidence, from a 1990 study in Health Education Quarterly — “Evaluating Understanding of Popular Press Reports of Health Research” — is not comforting. The overall rate of college-age reader misunderstanding was 40 percent, which suggests that the public is as likely to misinterpret accurate reporting as inaccurate reporting.
So, to sum up: You have the constant publication of new research, which may or may not be flawed (and distorted by its own PR) or incompletely reported by the media to the point of misrepresentation, which is then likely to be misunderstood by a significant amount of the public. The rational response is not just cui bono, but why bother?
