In this Linked-in article, “Are We All Being Fooled by Big Data?,” Michael Moritz of Sequoia Capital uses Nate Silver’s recent book The Signal and the Noise: Why Most Predictions Fail but Some Don’t as a launching point to discuss the limitations of predictive science.
Moritz makes some good points but, of course, the Bible was well ahead of him, saying in ancient times:
I again saw under the sun that the race is not to the swift and the battle is not to the warriors, and neither is bread to the wise nor wealth to the discerning nor favor to men of ability; for time and chance overtake them all. – Ecclesiastes 9:11
Here’s the first two paragraphs from Moritz:
When, on a summer Sunday morning in 1987, three hundred thousand people crammed onto the central span of San Francisco’s Golden Gate Bridge, they came perilously close to participating in the largest accident in American history. The bridge’s engineers had made copious calculations and had designed it to sway nearly 28 feet and shoulder the burden of hundreds of vehicles. But nobody had ever predicted that a gigantic crowd of pedestrians, attracted by the fiftieth anniversary of its opening, would be stuck between its towering pylons unable to move in any direction. As a result, the bridge flattened out and came within whiskers of straining every last fiber of its vermilion superstructure.
The consequences of faulty data, wonky forecasts, ill-conceived opinions, loose predictions, incorrect assumptions and, in the case of the Golden Gate Bridge, an improbable event form the backbone of Nate Silver’s absorbing new book, The Signal and the Noise: Why Most Predictions Fail but Some Don’t. This book, written by the voice behind the popular election forecasting blog, FiveThirtyEight, now licensed by the New York Times, is a reminder that while data doesn’t lie, it does allow people to deceive themselves and others. In some cases it’s a question of the bigger the data, the grander the deception.