If you know of John Ioannidis‘ work, Jonah Lehrer’s new piece in The New Yorker won’t be a surprise to you. It’s alarmingly titled The Truth Wears Off – is there something wrong with the scientific method? Here are some sections which you can’t get without a subscription, and I think they get to the heart of the problem:
“Whenver I start talking about this, scientists get very nervous,” he says….
Jennions admits that his findings are troubling, but expresses a reluctance to talk about them publicly. “This is a very sensitive issue for scientists,” he says. “You know, we’re supposed to be dealing with hard facts, the stuff that’s supposed to stand the test of time. But when you see these trends you become a little more skeptical of things.”
There is no mysterious “force” in the universe. The answer is probably going to come down to a combination of the reality of randomness (regression to the mean falls into this category), individual bias, and the cultural incentives of the system of scientific production. This is partly a coordination problem. Most social psychologists, to pick on one discipline which even other psychologists will finger-point toward, are probably aware that their results aren’t going to be robust over the long haul. But they have tenure to gain, mortgages to pay, and fame to accrue. This is not furthering the collective system-building which is science, but the first person to opt-out of rat-race for sexy findings which have publishable p-values will soon be an ex-scientist.
If you don’t have a subscription to The New Yorker, buying one off the newsstands for an article like this is much more worthwhile than another boring political profile. You should also check out Why Most Published Research Findings Are False. You can read that for free. Also see David Dobbs’ How to Set the Bullshit Filter When the Bullshit is Thick.
Note: Statistics are ubiquitous across many of the sciences, but the reality is that most people who use statistics don’t understand them too well. That’s not necessarily an issue, most people who use computers don’t know how they work, but then again, most people don’t use the mouse as a foot pedal.