When I wrote the short post Understanding Social Scientists and Their Role last week, I had no idea a published social science study would support my skepticism about social science – but here it is!
In the October 19, 2015 issue of The Weekly Standard, senior editor Andrew Ferguson writes a six-thousand-word article. That’s a lot to read, but here are the first five hundred words which will give you the gist of the point he wants to make.
One morning in August, the social science reporter for National Public Radio, a man named Shankar Vedantam, sounded a little shellshocked. You couldn’t blame him.
Like so many science writers in the popular press, he is charged with reporting provocative findings from the world of behavioral science: “. . . and researchers were very surprised at what they found. The peer-reviewed study suggests that [dog lovers, redheads, Tea Party members] are much more likely to [wear short sleeves, participate in hockey fights, play contract bridge] than cat lovers, but only if [the barometer is falling, they are slapped lightly upside the head, a picture of Jerry Lewis suddenly appears in their cubicle . . . ].”
I’m just making these up, obviously, but as we shall see, there’s a lot of that going around.
On this August morning Science magazine had published a scandalous article. The subject was the practice of behavioral psychology. Behavioral psychology is a wellspring of modern journalism. It is the source for most of those thrilling studies that keep reporters like Vedantam in business.
Over 270 researchers, working as the Reproducibility Project, had gathered 100 studies from three of the most prestigious journals in the field of social psychology. Then they set about to redo the experiments and see if they could get the same results. Mostly they used the materials and methods the original researchers had used. Direct replications are seldom attempted in the social sciences, even though the ability to repeat an experiment and get the same findings is supposed to be a cornerstone of scientific knowledge. It’s the way to separate real information from flukes and anomalies.
These 100 studies had cleared the highest hurdles that social science puts up. They had been edited, revised, reviewed by panels of peers, revised again, published, widely read, and taken by other social scientists as the starting point for further experiments. Except . . .
The researchers, Vedantam glumly told his NPR audience, “found something very disappointing. Nearly two-thirds of the experiments did not replicate, meaning that scientists repeated these studies but could not obtain the results that were found by the original research team.”
“Disappointing” is Vedantam’s word, and it was commonly heard that morning and over the following several days, as the full impact of the project’s findings began to register in the world of social science. Describing the Reproducibility Project’s report, other social psychologists, bloggers, and science writers tried out “alarming,” “shocking,” “devastating,” and “depressing.”
But in the end most of them rallied. They settled for just “surprised.” Everybody was surprised that two out of three experiments in behavioral psychology have a fair chance of being worthless.
The most surprising thing about the Reproducibility Project, however—the most alarming, shocking, devastating, and depressing thing—is that anybody at all was surprised. The warning bells about the feebleness of behavioral science have been clanging for many years.
Thus we now have a social science study which shows why we should be cautious about the pronouncements of social science studies. I have to figure the irony was lost on NPR.
I am emphasizing this point because in our day secular society thinks and acts as if this or that “scientific” study deserves more trust than the Bible. People can show you a survey that will make whatever political or social point they’re wanting to push (e.g. “Children Raised by Free-Range Panda Bears Fare Just as Well as Those with a Human Mom and Dad”). Don’t fall for it. The word of God and conscience are far more reliable guides.
As I quoted in the previous post, “There are three kinds of lies: lies, damned lies, and statistics.” Yet, as Paul said, “Let God be found true, though every man be found a liar” (Romans 3:4).
(24-minute read; 6,170 words)