Big data: the bite-sized lowdown

Reading through technology news and opinion articles today, it is pretty hard to avoid the term ‘big data,’ Harvard Business Review blog, Forbes, MIT Sloan Management Review, McKinsey Quarterly have all covered it extensively. As one analyst has commented, the hype about it is reaching “the levels of SOA in the early 2000s, cloud in the late 2000s, and social in the past few years.”

Yes the topic is hot and there is a lot of noise about it – but what exactly is ‘Big Data?’

Simply put, Big Data is: “extremely large volumes, of mainly unstructured data, which are streamed (not batched), and which require real-time analysis.”

The main characteristics of big data are described as:

• Volume — This original characteristic describes the relative size of data to the processing capability. Overcoming the volume issue requires technologies that store vast amounts of data in a scalable fashion and provide distributed approaches to querying or finding that data.
• Velocity — This describes the frequency at which data is generated, captured, and shared.
• Variety — This refers to the proliferation of data types from social, machine-to-machine, and mobile sources, adding new data types to traditional transactional data.

So who is using big data? Forrester Research indicates that the businesses using big data are the ones that have always handled lots of data in the past – e.g., banks and insurance companies, telcos, oil companies, large retailers, and the international CPG companies. An analysis by McKinsey defines those industries where big data has high value vs. those where its value is low, and also indicates that currently the value of big data isn’t that great for manufacturing businesses.

How relevant is big data for small and mid-sized businesses? It seems that most of the hype is coming from database vendors and consultants, but let’s put it in perspective and consider the following:

1. Most organisations don’t yet have the data sources for it – whether from customers, manufacturing plants, operations, or inventory.

2. The SMB market is very conscious of risk and cost – big data technology is currently ‘bleeding edge’ and the cost of implementation is high. There are also other issues – sorting out the technology isn’t even 50% of the work; there are new skills, processes, and management approaches that need to change to make proper use of it.

3. Many organisations that have large data volumes (in the gigabyte range) can easily handle them using existing relational database (RDBMS) software. The combination of existing Microsoft technologies – e.g., SQL Server 2012, Power View, and Power Pivot – provide the capability to manage and analyse large volumes of data without having to implement new technologies.

Considering the above, what could make big data become more important to the SMB market?

1. The data available gets above a certain critical threshold.

2. Measuring and recording instruments become a part of operations to generate very large volumes of data.

These conditions could arise with the growth of the Internet of Things. This refers to how objects are becoming embedded with sensors and linked through networks, often communicating via the internet. This could create huge volumes of data that flow to computers for analysis, and could therefore have an impact on SMB manufacturers and distributors.

But for now, don’t get too caught up in the hype, tread slowly and wait to see if it is in fact relevant and necessary for your business.