Ivan Oransky has a recommendation for readers tempted to click on the latest headline about chocolate, alcohol, or coffee curing every disease under the sun: just don’t read it.
“I guess if you live in France, fantastic,” says Dr. Oransky, distinguished NYU Journalism Institute writer in residence and MedPage Today editor at large. “If you like any of those things continue using them in moderation the way you see fit, and the way you and your doctor discuss. If you don’t like those things, don’t use them. Don’t be under any illusions that any of it has very much to do with how healthy you’ll be.”
One caveat, he says, is people who have struggled with alcoholism, who ought to avoid red wine altogether. And for all readers, the latest studies just aren’t that practical. “Reporters and editors love those stories, and they’ll probably continue loving those stories,” Dr. Oransky says. “But they’re just not useful.”
Dr. Oransky, who founded both Retraction Watch and Embargo Watch and is vice president of the Association of Health Care Journalists, has made a career of focusing on under-reported factors that shape — and block and distort — the flow of scientific information.
“Embargoes have been stuck in my craw for a long time, probably for as long as I’ve been aware of them,” he says. When he read Vincent Kiernan’s book “Embargoed Science,” which argued that journals place a stranglehold on the flow of scientific information, Dr. Oransky found it convincing. In November 2007, he reported for the Scientist about the World Health Organization punishing the New York Times for breaking an embargo. “It was a little bit like detention, or no recess break,” he says.
“I’m sure the Times will figure out a way to report on the WHO without the embargoed material. They’re a newspaper. It’s what they do,” he wrote back in 2007. “Two weeks isn’t that long a period of time. What struck me was the public flogging.” (Jack Shafer quoted that last phrase when he reported for the matter in Slate.)
In February 2010, Dr. Oransky founded Embargo Watch. “My hope is that by chronicling these stories and trends, I can help make embargoes work better,” he wrote in the inaugural post, “Why write a blog on embargoes?”
“As I would call some of these journals to the carpet for doing things that I thought were really not kosher, they stopped. Some of them would issue statements saying, ‘We’re not going to do that. That wasn’t the right thing,’” he says. But Dr. Oransky has seen slippage as he’s had to mostly put his embargo blogging on the back burner. Where close-hold embargoes, which generated much discussion about six months ago about the U.S. FDA, were also an issue in 2011, when Dr. Oransky covered them.
“Over the past couple of years, where I haven’t been blogging much on Embargo Watch, I’ve seen a lot of these behaviors creep back up. It makes me a bit sad,” he says. “Journals are going to do what they’re going to do, and it’s pretty clear they’re doing it for their own interests. I wish that weren’t the case.”
The role embargoes are playing isn’t just academic. “Writ large, the average reader is not getting a realistic picture of science,” Dr. Oransky says. “They’re not getting the whole narrative. They’re basically getting shiny, happy, promotable bits, and they’re not getting what’s actually happening.”
And then there’s the subject of Dr. Oransky’s second study: retractions, which he calls the “nuclear option” of correction. Instead of just a mistake that bears correcting, a retraction of a study suggests that the journal staff believes that the mistake or misconduct — the latter accounts for two-thirds of retractions — was so egregious that the staff no longer trusts the paper anymore. It’s a good thing, therefore, that papers are being retracted.
“It’s a bad thing for that paper,” Dr. Oransky says. “But it means that somebody is paying attention. It means that science is, at least in some cases, trying to correct itself.”
The number of retractions has grown significantly since 2000, and over that span, the rate has grown about 20-fold. In 2000, it was about 35-40, whereas it is now between 800 and 900, and the number of papers published in 2010 was 44 percent greater than in 2000, so retractions are really growing at a faster rate, Dr. Oransky says,
Although most retractions occur in basic science — either in mice or test tubes — the studies inform clinical trials, which therefore inform practice. Some occur in patient-studies, Dr. Oransky notes, and the two current “leaders” of Retraction Watch’s leaderboard are anesthesiologists. (One has a whopping 183 retractions, and the other 96.) A third anesthesiologist made up all of the patients in a study, so no patients were actually harmed by the misconduct, although subsequent patients would be impacted if they were treated based on the sham study.
“This does matter,” Dr. Oransky says, of the retractions, which he notes are a small part of the broader picture. “It doesn’t mean you shouldn’t take treatments. It doesn’t mean you shouldn’t get involved in trials. Quite the opposite,” he says. “It means you should understand what is the evidence behind them. If some of that evidence turns out to be retracted, one of the problems is that a lot of these papers continue getting cited. That’s a problem.”
“If someone walked into a courtroom and said, ‘Hey your honor and members of the jury. Here’s a precedent that I think means that my client is not guilty, or should be let go. If it turns out that precedent had been overturned by a higher court, that’s not going to be a good outcome for that attorney and for his or her client,” Dr. Oransky says. “That’s not true in science. You can keep citing retracted papers. There’s no real consequence.”
Evidence shows that fraud is on the rise in certain fields, which likely means it’s happening everywhere, Dr. Oransky says. He notes that “publish or perish” is a real thing, and most scientists are judged primarily by what and how often they publish.
“We have not caught all the papers that should be retracted yet. That’s for sure,” he says.
Retractions, however, present very binary ways of looking at the world, and things weren’t always that way. “We all act as if peer review is this thing that was created along with whatever your favorite myth is. It wasn’t,” Dr. Oransky says. “Even until the 50s, 60s, and 70s, peer review did not happen the way we think about it now. Watson and Crick wasn’t peer reviewed. Last time I checked, it’s still true. And it’s probably one of the three most important papers ever.”
Six months prior to James Watson and Francis Crick’s discovery of the double helix, Linus Pauling rushed his own publication of misguided theory of a triple helix to beat the two to the punch. “It’s a great nugget of a story. He knew that they were kind of getting somewhere with this and thought they were going to publish,” Dr. Oransky says. “He wanted to get out there first and be right, even though he turned out to be wrong.”
In 2012, Retraction Watch ran a post about whether Pauling’s paper should be retracted, and Dr. Oransky was surprised by the “venom” of the comments. (One said, “It appears this is another rubbish site that spins a piece of no-news that is not useful and absolutely boring to boot.” Another added, “I won’t read RW again until this post is retracted.”)
“That was pretty amazing to me, because it reminded me that science is a guild, like any other, and when you question people in that guild they tend to circle the wagons,” Dr. Oransky said. It also made him think that sometimes it’s best to include a note on a disproved paper, rather than issuing a retraction.
So what should readers do when they want to know if a study is reliable? Here’s what Dr. Oransky suggests:
- Check the retraction database, which isn’t quite comprehensive yet, but Dr. Oransky says is nearing completion. The database is searchable by paper title.
- If you’re reading about a study in the news, the chances are good that it’s a new study. “It wouldn’t be retracted right when it came out,” Dr. Oransky says. “That would be weird.” Instead, consider whether the article accurately reflects the study, and whether the study accurately reflects reality.
- The site HealthNewsReview, which bills itself as the “health news watchdog,” is a valuable and reliable resource, Dr. Oransky says.
- When determining if a source reliable, see if it has a corrections policy. “If they don’t publish corrections, I actually don’t trust them,” Dr. Oransky says. “If they have an active corrections page, I do trust them.”
- Make sure the publication names names. “If you’re reading about a study, and they don’t name the journal, the authors, and the university, and they don’t link to the study, that’s just not a good sign,” he says. “Often that means that they are trying to tell you something is bad for you, so that they can sell you vitamins.”
- Although Dr. Oransky always encourages people to read primary sources, it’s not necessarily a great use of time to read every new study. “In the long run, you’ll just chase your tail reading a lot of studies,” he says. Instead, consider reading meta-analyses, which conduct studies of studies. “They’re going to gather all the high-quality evidence and tell you if it’s high quality,” he says.
- A culture where many feel the need to react immediately to everything, isn’t setting itself up for the best decision-making. “If you want to make the best decisions about your health, read the summary, read everything that has come before and understand the context,” Dr. Oransky has says. Even if each new study is correct, “it may only be one star in the sky.”
Additional reading:
- Richard harris’ new-ish book, “Rigor Mortis.”
- Christie Aschwanden FiveThirtyEight article “Science Isn’t Broken”
- Oransky’s Dec. 2015 article “How Publish or Perish Promotes Inaccuracy in Science–and Journalism” in the AMA Journal of Ethics.
