You hear it all the time – the rate of change is accelerating, and it’s almost impossible to keep up. But here’s the problem – we usually make the situation worse, and it’s totally unnecessary. We dramatically over-react to the latest buzzword and at the same time miss the opportunity to revisit the fundamentals that we haven’t adequately applied to begin with.
Take, for example, the “Big Data” trend. What did this term really mean? Data was growing in volume and complexity. New technologies for processing and managing data were being implemented more widely. But these trends had been under way for decades. Yet, strangely, it was the introduction and popularity of the term “Big Data” itself that prompted the urgent questions from business and IT leaders everywhere: “What do we need to do? How should we exploit this – whatever it is? How do we respond? What do we need to change?”
There have been three possible responses to this:
Option 1: Take this as an opportunity to finally establish a comprehensive and rational data strategy including a plan to build the appropriate business-driven technical architecture, organization, and integrated data, leveraging new technologies where it makes sense.
Option 2: Attempt to create a narrowly-focused “Big Data Strategy” (or “Data Lake Strategy”) covering the same ground, while ignoring the fact that there is still no effective strategy, architecture, or organization in place for data generally (hint: this option is completely irrational).
Option 3: Having a solid strategy already in place, carefully examine the genuinely new opportunities available with Big Data and decide how to take advantage of those opportunities.
Which do you think was the most common choice? I’d say Option 2 by a long mile. Why? I don’t know – maybe Option 1 seems hard and doesn’t sound very sexy. And Option 3 is not really an option if we never buckle down and do Option 1.
What was the result? Well, Gartner had predicted that through 2018, “90% of deployed data lakes will be useless.” Here we are at the beginning of 2018. Are we close that result? Probably.
What did we do wrong? I would argue the root causes are timeless, as are the solutions. And the deepest and most central root is strategy.
The right way to develop a data strategy is:
Step 1 – Identify the top business initiatives of the organization, including any new initiatives to take advantage of emerging technologies.
Step 2 – Identify the data and analytics needed to support the initiatives, including shared data opportunities across and within initiatives, as well as supporting infrastructure, leveraging a range of technologies and the strengths of each.
Step 3 – Identify the enabling capabilities needed, including data management, governance, organizational structure, process, and so on, including a capability for experimentation and exploration as well as production.
Step 4 – Assemble the identified needs into a roadmap – a coherent plan of action – and implement the roadmap in incremental stages, through projects or an agile program approach.
Although many organizations attempted to create a “Big Data Strategy”, the steps above were typically not followed. For example, dumping a bunch of data into a data lake and asking smart people to curate that data for everyone is not a strategy. And guess what? These steps have usually not been followed for data generally – “big” or otherwise – and this continues to be at the root of the challenge with enterprise data and analytics more broadly. That’s the missed opportunity.
Are these seemingly simple steps passé? If someone can convince me that we don’t need to support business initiatives anymore or that a coherent data and analytic ecosystem is no longer desirable or valuable, then I’d say yes. But so long as data is stored and analyzed for business use, there will be a need to organize and use it rationally. It’s that simple.
Almost every new buzzword that emerges presents yet another opportunity to develop a comprehensive and effective data strategy, including a vision and plan for a coherent ecosystem and organization. It’s an opportunity re-visit the fundamentals – or to finally visit them for the first time.
Which new trend will present us with the same options yet again?
It’s true that change is increasing, and the pace of change will continue to accelerate. The good news is that we can get control of and exploit these changes by embracing timeless principles that work.
Kevin M Lewis is a Director of Data and Architecture Strategy with Teradata Corporation. Kevin shares best practices across all major industries, helping clients transform and modernize data and analytics programs including organization, process, and architecture. The practice advocates strategies that deliver value quickly while simultaneously contributing to a coherent ecosystem with every project.
View all posts by Kevin Lewis