“Big Data” is a relative term and means different things to different organisations. It is usually not about the volume of data, but the approaches to both data collection and storage and the use of that data to realise business benefits. The following three signs can be useful to indicate if an organisation is entering “Big Data” territory:
- Existing IT infrastructure cannot cope with the growing dimensions of data in a cost-effective manner.
- Organisation cannot achieve its objectives without analysing a broader range of data.
- Benefits can be achieved by the application of new analytic techniques (such as machine learning) that go beyond traditional BI and statistical tools
Zuhlke Engineering Ltd with our partners BigData4Analytics and Hortonworks recently brought a group of senior business and technology leaders together for a breakfast Round Table to share their thoughts and experiences on “Big Data” and what it means to them. Participants’ backgrounds ranged from Financial Services to Retail, Telco and Charity organisations.
The discussions covered a number of topics, and the key takeaways were:
Vision and Strategy questions remain for many companies. The UK and EU are generally lagging behind the US in terms of adoption, although this varies significantly by sector. Finance, Retail and Telco have been quicker to adopt new technologies and approaches to analytics, used as they are to collecting and processing large volumes of transactional data whilst other industries such as construction and manufacturing lag behind. Take up has been hampered by low Board level awareness and traditional mindsets coupled with skills shortages and fears over security – too often departments are reluctant to share data and adopt new strategies to explore potential benefits.
The great advances that have been made in analytics and the availability of open source algorithms now allow even the smallest companies to interpret past events to “best guess” the future – predictive analytics are therefore being used in more and more contexts. The application of analytic capabilities combined with the increased scope, content and context of Big Data, particularly when merged with more traditionally structured datasets (such as a data warehouse or online analytical processing [OLAP]), has drastically increased the variety of use cases for decision support, and in some cases, decision automation. For companies like Amazon, this means already beginning to ship customers’ packages before they even order them, giving the retailer an unparalleled advantage in shipping speed. Today, the value of predicting the future of markets and consumers is on the rise, based firmly on computer science research and using the data from both internal and external sources.
Technologies such as Hadoop are not going to replace the traditional Data Warehouse any time soon. However, they will be deployed more often to augment existing solutions and enable innovations that use more advanced analytic techniques with smaller and smaller latencies i.e. the time from data collection to data based decision. The Data Warehouse will remain to support operational analytics and statistics, but is generally less cost effective for storing and processing varied and unstructured data which is Big Data’s strength.
There are many challenges that hinder getting “Big Data” projects off the ground. These include identifying the decision makers, working out how to measure ROI and over what timescale, understanding whether the benefits are direct (such as increased sales or reduced costs) or indirect (increased productivity, better customer engagement…), deciding how big or small to start, obtaining and cleaning the data and determining meaningful KPIs to measure success. The breakfast participants advised to start with a business problem first – not a technical solution. Demonstrate the potential for tangible value to the organisation and get senior management buy-in. Finally remember that first projects are just the beginning of the “data journey” – not the destination.
In summary, the name Big Data is not really appropriate. It is all about innovative use of data to obtain both Insight and Foresight for advantage. What is certain is that the worlds of Big Data, Machine to Machine and the Internet of Things are the next revolution changing the way we work and interact in our daily lives!