Taming the Data – A CIO’s perspective
The Business of managing the data & getting value out of it is a fast-changing one. Currently, this space is bubbling with a plethora of large and specialist product vendors offering a range of offerings marketed in an almost continuous stream of conferences & summits aiming to evangelize towards their product’s direction. CIOs exposed to these events often come back with useful knowledge of how the Digital natives are driving the value of data. However, they still look for a clearer path to transform their own Enterprise Data landscape. The aim of this short article is to provide CIOs our view on some of the key choices they need to make in this journey.
One of the most important choices that a CIO will make is how to organize define the Organization structure to lead & support the data initiatives. In large enterprises, it is common to come across multiple functional departments, regions, and product lines running their own data platforms. Very often, each of the teams has its own technology platform choices. Many CIOs typically create centralized Data teams & often bring in a CDO or a Head of Enterprise Data to manage this vision of data-driven transformation. The breadth and remit of this Enterprise data team will significantly determine the transformation path. In our experience, we have seen CIOs experiencing the biggest success where they chose a middle path. The Central Enterprise data team /CDO Organization is responsible for planning, designing, defining standards & governing an Enterprise-wide Data platform capability. These enterprise-wide data platform capabilities are in-turn leveraged in the Departmental or regional Data Initiatives.
This Enterprise Data platform should not be equated with an Enterprise-wide Data Lake but as a set of capabilities in terms of Infrastructure, frameworks, tools, catalogs that will help drive the Time to Reliable Insights metric across the Enterprise.
In our experience, too often the Central Enterprise Data teams single biggest ask from the CIOs is to fund an Enterprise-wide data lake initiatives. These large high profiles monolithic programs with the objective of a single repository of Enterprise-wide data gives a false sense of purpose & promises to be the singular deliverable supporting the CEO’s data-driven transformation ambition. These large monolithic Programs run by central teams often struggle to handle the complex data domains as well as the complexity of the Business use cases that fund these initiatives. Lacking the depth of the domain data understanding, these teams often end up producing dumb copies of the data without the data domain caress & quality check that is typically required. This leads to a delay in delivering the use cases & business value.
Enterprises that are more successful have benefited from a microservices type approach where the Central Enterprise data team provides the Platform, tools & frameworks but the projects themselves are use case focused and delivered with a Product centric approach combining cross-functional expertise from various teams. A large Airline under the leadership of its CDO has built a Centralized Data platform offering a host of capabilities like Automated Infrastructure provision, Polyglot shared storage, Automated & self-service pipeline generator, Schema registry, provision to secure data access, data discovery and cataloging mechanism, etc. Individual projects run in a Product centric approach either by the Business units or by the Central teams, as the use case demands.
In our experiences, CIO s who have followed this direction has seen the biggest success in their data initiatives. Using this approach they have been able to leverage the benefits of a modern data platform and at the same time successfully manage the change by co-opting the departmental initiatives in this common vision.
Another common trend we see in the Industry is Enterprises wanting to build a number of low-level Data platform building blocks. They typically like to follow the incredible innovation in data happening in digital-native companies like Netflix, Uber, Pinterest, Zalando. This means wanting to build a number of low-level data components rather than adopt what is largely available in the market place. For e.g. a Head of Data Platforms for a large Fortune 100 Pharma Enterprise spent more than 10 months to build a bespoke Enterprise-wide standardized Ingestion framework. This led to delays in the speed of the overall Data on cloud transformation that the CIO originally started to embark on. In another example, a large Insurer has been building a bespoke Metadata driven ETL framework for a popular cloud-native Data warehouse platform for over 8 months. In our opinion, often these projects can be fast-tracked by building upon available frameworks & tools in the market.
Often these bespoke framework-building engagements are set up to satiate the engineering talent in the Organization. Unless your business is ready for this, CIOs can do their organizations a favor by questioning the value of these Build on your own projects.
We have seen few Enterprises increasingly adopt Event-driven microservices architectures for their Digital programs. A Central Event-Driven data hub is often a central component in these initiatives. This is also aided by the increasing maturity of platforms like Kafka. A common use case where Enterprises are investing in is a customer event hub to combine customer interactions across different channels to provide a single view of the customer information. Having seen success in driving better customer engagement, they are emboldened to carry this forward as a central theme across the enterprise. We see a big potential for CIOs to evolve their Data platforms as event-driven modern Data Hubs forming the center of the Digital Infrastructure. These event-driven data hubs can act as the forefront of the Organization’s digital transformation and can be a truly data-driven enterprise.
Data catalog & Data discovery - While self - service BI has been a point of discussion in this space led largely by the BI products like Tableau, Qlik & Power BI, the maturity of data catalog tools have now allowed Enterprises to build the critical data discovery layer to support these initiatives.
Data Marketplaces - Easy Data sharing & collaboration among partners has become another new use cases for DW modernization using cloud DW platforms like Snowflake. A leading CPG manufacturer has replaced its complex data sharing processes built on FTP and custom
Data Lakes & Data warehouses – We see a lot of confusion in the market aided by many product vendors now claiming to possess a unified Data platform for housing the Data Lake and the Data warehouse in a seamless Architecture –aka Data Lake Houses. While this is a positive trend, we advise the CIOs to look at this as a capability that will mature only in couple of years. For now, there is a distinctive place in an Organization for a Data Lake and a well-defined DW & data marts catering to BI Dashboards & SLA driven management reporting.
