Big data is invaluable to society – the majority of businesses now understand its importance, even more so with the global events of the recent months, with many now prioritising time and resources to transform raw information into transformative insight and competitive advantage. However, where a significant number of companies fall down is the techniques required to best monetise their data to gain insights and advantage.

This is perhaps understandable given the accelerating pace of innovation across the subject, bringing both opportunities and risks to those organisations looking to exploit big data to find the trends, patterns and outliers hidden deep inside their data to help accelerate digital transformation initiatives. Indeed, as set out in a recent report, “…the big data industry is experiencing profound changes across the entire stack including infrastructure, security, analytics, and the application layer…shifting from host-based network topologies to cloud-based, data-centric architectures, thereby creating enormous challenges and opportunities for transitioning and securing data systems.”

What’s the big idea?

Historically, large companies have not only had the most data to analyse, but they have also had the resources required to address the issue. That said, as technology evolves, big data analysis and management is no longer reserved for large corporations – mid-sized companies can profit from this type of technology with bespoke tools that fit their company size to better compete with the giants and challengers alike.

In practical terms however, many companies risk falling at the first hurdle by failing to understand the data they have, as it remains in siloed compartments instead of in a comprehensive and integrated solution. This ultimately affects all areas of a business, from the potential to create new business models, to improving a customer engagement strategy in order to engage with customers in new and meaningful ways, in order to build differentiated services and solutions. But, determining which data sets can provide value to the business – a simple problem in theory – is often more difficult to pin down in reality. The sheer volume of data to examine can be daunting and finding the ‘gems’ can be a challenge.

These large data sets are gaining in importance and have catapulted big data to a new level of complexity and scale. There is a recognition that a single ‘data lake’ strategy simply will not work, as invariably they become a ‘data swamp’ – different data types have different characteristics and needs. In overcoming these issues, the fundamental point to recognise is that a big data strategy starts with a data management strategy.

Getting your ducks in a row

An effective data management strategy should focus on exactly how data is handled from the moment it is created – right from the very edge of the corporate network to the data centre. To ensure the strategy delivers, businesses must understand the value of their data: does it need to be retained, for instance, and if it does, how long for? Where should it be stored so it can be easily accessed and used? If it doesn’t need to be retained, what needs to happen to it?

The key to data democratisation is the ability for all data to be available to everyone with the relevant permissions and access controls.

Retention is an important issue and should be based on business value as well as ensuring compliance with applicable legislation. In addition, the correct levels of accessibility and management of data need to be included – data retention rules and principles should enable the creation of appropriate processes to identify important data and then manage it in the most efficient and cost-effective way in the long term.

Choosing an appropriate data management platform should be based on its ability to support the data management strategy. Organisations should actively engage with data management and storage service providers in order to improve their data management, data quality and data governance. This underpins compliance and lays a solid foundation for organisations to leverage value from their data through analytics and intelligence to the monetisation of those assets.

A centralised, compliant data management platform with proper checks in place can ensure that data is correctly captured and categorised, making it available to businesses in order to derive the insights that accelerate business transformation. The more sophisticated solutions can also analyse data and determine its value to the business. This enables users to identify what to keep and what can be discarded, translating into cost savings as businesses do not need to purchase and manage unnecessary capacity for superfluous data. Ultimately, bringing the right processes, partners and management technologies together can help unlock the huge potential of big data to deliver long-term impact, helping organisations transform, modernise and engage with customers in new and meaningful ways.