For our third flagship report on the State of Big Data 2020, we asked data strategists, architects and users what is driving investment in Big Data.
There have been two key developments in the past year that have marked the course of the Big Data industry: the mergers and acquisitions that took place throughout 2019, and the more recent coronavirus outbreak.
These two developments resulted in many companies now rethinking their operation, while also focusing on keeping the lights on. This is evident in the year-on-year changes we see in what Big Data professionals view as the main drivers for Big Data investment in 2020. IT Modernisation (30.8%) remains at the top of the list of key drivers of Big Data investment, with initiatives that can help cut costs (23.1%), and increase sales (20.5%) also taking priority.
For any company, the IT department is at the heart of digital transformation. However, managing large numbers of disparate, often legacy technologies and custom code is becoming increasingly difficult. IT budgets are consumed by resources, internal and external, as well as software and hardware maintenance fees. At the same time, delivery timelines are extending. And by the time many IT solutions make it to market, the needs of the business have often changed, rendering them obsolete on arrival or in need of even more development.
For any business, transformation should start with IT. Big Data technologies can help IT departments rethink their role, rationalise their ecosystem, and optimise their processes whilst acquiring new skill sets.
There are 3 main areas where Big Data can have a positive impact on costs: by detecting patterns and anomalies, by automating processes, and by reducing the dependence on legacy/high-TCO infrastructures.
Big Data analytics allow businesses to analyse large data sets, even in real-time, to derive meaningful conclusions. For a manufacturer, an unnoticed anomaly in energy consumption can lead to thousands lost in energy waste. For a retail chain, missed changes in customer behaviour may lead to excess inventory and the many costs associated with it.
A lot of business processes are built around collecting, analysing and reporting on data. As manual processes, they can be resource-intensive, prone to human error, and as a result, costly. Big Data analytics, AI and machine learning can automate different processes as diverse as detecting fraud, recognising voice, or analysing customer spending patterns, saving the business time, effort and money.
Companies that have been around for 5-10 years or more, may have accumulated over a thousand systems and applications, many of which are on-premises and quite a few have probably reached end-of-support/end-of-life. Big Data solutions, especially those hosted on the cloud, can help minimize legacy systems’ vendor support and maintenance costs, as well as on premises data infrastructure costs, like upgrades and support, electricity, storage and IT staff.
Legal & regulatory requirements are the next big driver (17.9%) as the sheer volume of data being produced raises concerns around secure data-processing. The investment in Big Data technologies goes hand-in-hand with new mandates and regulations that must be followed to prevent negative financial and reputation implications for investing organisations.