Big Data loses its hipster shackles, is set to go mainstream

The term Big Data has been thrown around pretty willfully over the past couple of years. It makes sense too, after all, petabytes of data are being gathered everywhere in the digital age at a furious rate, and by a variety of devices. You might as well use that data. Until recently however the only people really trying to use that data effectively were a handful of startups.

Now though, it looks like Big Data is about to go mainstream.

Websites and devices that are collecting data on a large scale include ubiquitous information-sensing mobile devices, cameras, microphones, RFID readers, and wireless sensor networks. Most large corporates are sitting on mines of data by virtue of their everyday operations and vast networks. Social networks have also become information gold mines, generating massive amounts of user data.

According to a worldwide survey of IT leaders conducted by tech research company Gartner, 42% of respondents stated they had invested in big data technology, or were planning to do so within a year.

“Organisations have increased their understanding of what big data is and how it could transform the business in novel ways. The new key questions have shifted to ‘What are the strategies and skills required?’ and ‘How can we measure and ensure our return on investment?'” says Doug Laney, research vice president at Gartner. “Most organisations are still in the early stages, and few have thought through an enterprise approach or realised the profound impact that big data will have on their infrastructure, organisations and industries.”

The big data initiatives undertaken by companies are happening in a rapidly shifting technological landscape with disruptive forces that produce and demand new data types and new kinds of information processing.

Gartner says these companies are turning to Big Data technology for two reasons: necessity and conviction. Big Data is becoming critical because it has obvious or potential business opportunities that cannot be met with traditional data sources, technologies or practices. In addition, media hype is often backed with rousing use cases.

“This makes IT and business leaders worry that they are behind competitors in launching their big data initiatives. Not to worry, ideas and opportunities at this time are boundless, and some of the biggest big data ideas come from adopting and adapting ideas from other industries,” says Frank Buytendijk, research vice president at Gartner. “Still, this makes it challenging to cut through the hype when evaluating big data technologies, approaches and project alternatives.”

Despite these challenges, Gartner predicts that by 2015, 20% of Global 1 000 organisations will have established a strategic focus on “information infrastructure” equal to that of application management.

The research company says that companies in a wide variety of industries are provisionally collecting and storing a burgeoning amount of operational, public, commercial and social data in anticipation of being able to make use of Big Data technologies.

In most industries — especially government, manufacturing and education — combining these sources with existing underutilized “dark data” such as emails, multimedia and other enterprise content often represents the most immediate opportunity to transform businesses.

Although most of the big data hype is about handling the sheer size and speed of data available, Gartner reckons that the ultimate wins will be from those making sense of a broadening range of data sources.

“Business and IT executives regularly say that information is one of their company’s greatest assets,” says Laney. “Businesses are increasingly managing and deploying information more effectively than ever, but certainly not with the well-honed asset management discipline applied to their traditional material, financial or other intangible assets. The application of formal information valuation models will allow IT, information management and business leaders to make better-informed decisions on information management (IM), enrichment, security, risks, purchasing, collection, usage, bartering, productisation and disposal.”