The excitement about “big data” in tech circles is very optimistic and many companies are rushing to hire “Data Scientists” to profit from the explosion of hype about the reams of data collected inside their own organizations, and in the world outside.
No ad to show here.
But having access to big data doesn’t guarantee that companies, or individuals, will understand or be able to derive much value from it. The very few examples of companies doing that, are very few. And for a good reason – finding insight in all that data is difficult and becomes more difficult the bigger the data sets.
Take for example the field of economics — it’s the original big data profession. But in all these years, it hasn’t been able to do much at all. The profession is well-regarded and respected despite its collective failure to understand the economy and predict its behavior.
Surely, a big data profession such as the study of economics over the past 150 plus years would by now be refined and almost scientific in its precision, especially since these days we have as much compute power as an economist might need, not to mention even more data to analyze. But it’s not even close.
After the financial meltdown in 2008, Alan Greenspan, the former Chairman of the Federal Reserve was asked questions by a Washington committee about how the crisis occurred.
He said all his financial models over the past 40 years were wrong. Yet those models informed his adjustment of interest rates, and if they were based on wrong models, he likely harmed the economy, consistently, decade after decade. It’s truly an epic fail for big data.
It doesn’t require much Googling to discover more examples of the deceit of economists and the failure of Big Data:
It’s hard to believe now, but not long ago economists were congratulating themselves over the success of their field. Those successes — or so they believed — were both theoretical and practical, leading to a golden era for the profession.
Economics is easily the single most important failure of the application of Big Data. And to call economics the “dismal science” is unfair on scientists because there’s nothing scientific about it.
How will our new workforce of data scientists fare in their missions? I predict dismally. It’s better they avoid Big Data and focus on the small data — just the bits that matter.
This article by Tom Foremski originally appeared on Silicon Valley Watcher, a Burn Media publishing partner.
